Home /Tools /AI Crawler robots.txt Generator

AI Crawler Blocker

Generate a custom robots.txt snippet to easily prevent popular AI crawlers from scraping your content for model training.

Select Crawlers to Block9 / 9

OpenAI / ChatGPT GPTBotUsed by OpenAI to crawl data to train AI models.

OpenAI Search OAI-SearchBotUsed for search features in ChatGPT.

Anthropic / Claude Anthropic-aiUsed by Anthropic to collect training data.

Google Extended Google-ExtendedUsed to train Google's Gemini models (does not affect Google Search).

Common Crawl CCBotMassive open dataset used by many AI models (including LLaMA).

Cohere cohere-aiUsed by Cohere for training models.

Meta / Facebook FacebookBotUsed by Meta for various purposes including AI training.

Amazon AmazonbotAmazon's crawler used for Alexa and AWS AI.

Perplexity AI PerplexityBotAI search engine crawler.

Target Directories

Comma-separated list of directories to disallow. Leave as / to block the entire site.

Generated snippet

robots.txt

# -------------------------------------
# Block AI Crawlers (Generated by SiteTidy)
# -------------------------------------

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: Anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Why block AI crawlers?

AI companies often crawl the public web to gather massive datasets used to train Large Language Models (LLMs) like ChatGPT, Claude, and Gemini. If you want to opt-out of having your content used as training data, you can instruct these bots not to crawl your site using the standard robots.txt file.

Will this affect my search engine rankings?

No. The crawlers used for AI training data (like GPTBot and Google-Extended) are separate from the crawlers used for standard search engine indexing (like Googlebot and Bingbot). Blocking AI crawlers will not affect your visibility in traditional search results.

Is robots.txt strictly enforced?

robots.txt relies on the "honor system". Legitimate companies like OpenAI, Google, and Anthropic publicly state that they respect these rules. However, rogue scrapers or smaller AI startups might ignore your robots.txt file entirely.

Related Guides

Deepen your understanding with our expert articles.

Understanding SPF, DKIM, and DMARCRead guide →The Ultimate Guide to Meta Tags for SEORead guide →