Robots.txt Tester
Simulate search engine crawlers and test if your paths are accessible.
Robots.txt Tester
Test if a specific path on your website is blocked by robots.txt for a given user-agent.
How does the tester work?
This tool fetches the live robots.txt file from the root of the domain you specify. It parses the rules based on the Robots Exclusion Protocol, and determines if the given User-Agent is allowed to crawl the specified Path.
Common User Agents
- Googlebot: The primary crawler for Google Search index.
- Bingbot: The crawler for Microsoft's Bing search engine.
- GPTBot: OpenAI's crawler used to train large language models (like ChatGPT).
- CCBot: Common Crawl's bot, which archives the web.
- *: The wildcard user agent, which applies to any bot that doesn't have a specific rule set up for it.
Why is a URL blocked?
If the tool shows that your URL is blocked, it means there is a Disallow rule in your robots.txt file that matches the URL path you entered. The tool will highlight the exact rule that caused the block. To fix it, you will need to edit your robots.txt file to remove or override that rule with an Allow directive.
Related Guides
Deepen your understanding with our expert articles.