Robots.txt Generator
Quickly generate a robots.txt file to guide search engine crawlers.
Robots.txt Studio
Write, validate, and test crawling directives against specific bots in real-time.
What is a robots.txt file?
A robots.txt file is a simple text file placed in the root directory of your website (e.g., https://example.com/robots.txt). It tells search engine crawlers (like Googlebot) which pages or files they can or can't request from your site.
How does it work?
The file uses a standard called the Robots Exclusion Protocol. It consists of two main parts:
- User-agent: This specifies which crawler the rule applies to.
*means it applies to all crawlers. - Disallow/Allow: This specifies the paths that the crawler is not allowed (or allowed) to access.
For example, Disallow: /admin/ prevents search engines from crawling anything in the /admin/ directory.
Security Warning
Important: A robots.txt file is publicly visible. Do not use it to hide sensitive information, as bad actors can read it to find your private directories. If you need to keep a page private, use password protection or authentication.
Related Guides
Deepen your understanding with our expert articles.