Free tool
Control which AI crawlers read your site.
Toggle each 2026 AI crawler on or off, pick a global default, add your sitemap, and copy a ready-to-deploy robots.txt. The default stance keeps you in AI answers and blocks only scraper-only bots.
A robots.txt for AI tells each crawler whether it may read your site. Every provider runs a separate search bot and training bot, controlled by separate tokens. To stay visible in AI answers, allow the search and citation bots; block only scraper-only crawlers like CCBot and Bytespider.
robots.txt is a request, not a wall.
Reputable crawlers obey it. Others ignore it. Block a search bot and you remove yourself from AI answers; trust it to stop a scraper and you will be disappointed. The toggles below default to the stance that keeps you visible.
Control which AI crawlers read your site.
Toggle each 2026 AI crawler on or off, pick a global default, add your sitemap, and copy a ready-to-deploy robots.txt. The default stance keeps you in AI answers and blocks only scraper-only bots.
Free. No signup. Permissive by default.
Crawler controls
Each provider runs separate bots for search, citation, and model training. Allow or block each one independently.
- OAI-SearchBotOpenAI / Search / citation
OpenAI search and citation crawler. Allowing it is how you appear in ChatGPT search answers.
- ChatGPT-UserOpenAI / User-triggered
Fetches a page when a ChatGPT user follows or asks about a link.
- OAI-AdsBotOpenAI / Ad pages
Reads pages tied to OpenAI ad placements.
- GPTBotOpenAI / Training
OpenAI training crawler. Feeds model training, not live answers.
- Claude-SearchBotAnthropic / Search / citation
Anthropic search and retrieval crawler for Claude answers.
- Claude-UserAnthropic / User-triggered
Fetches a page when a Claude user follows or asks about a link.
- ClaudeBotAnthropic / Training
Anthropic training crawler. Feeds model training, not live answers.
- PerplexityBotPerplexity / Search / citation
Perplexity search and retrieval crawler. Allowing it is how you get cited in Perplexity.
- Perplexity-UserPerplexity / User-triggered
Fetches a page when a Perplexity user follows a citation.
- Google-ExtendedGoogle / Training
Controls use for Gemini training and grounding. Separate from Google Search indexing.
- Applebot-ExtendedApple / Training
Controls use of your content for Apple model training. Separate from Siri/Spotlight indexing.
- CCBotCommon Crawl / Training
Common Crawl. A third-party scraper whose archive trains many models. No search benefit. Blocked by default.
- BytespiderByteDance / Training
ByteDance training crawler. Notoriously ignores robots.txt, so a WAF is the real control. Blocked by default.
- Meta-ExternalAgentMeta / Training
Meta training and AI crawler (also seen as FacebookBot).
- cohere-aiCohere / Training
Cohere training crawler.
- YouBotYou.com / Search / citation
You.com search crawler.
The rule applied to any crawler not named above (the User-agent: * group).
Added as a Sitemap: directive. Must be an absolute https:// URL.
Your robots.txt
Updates as you toggle. Copy it, or download the file and place it at the root of your domain.
Read this before you deploy.
robots.txt is a request, not a wall. Knowing its limits keeps you from blocking yourself out of AI answers or trusting it to stop a bot that ignores it.
01
Tokens are case-insensitive
Crawlers match their user-agent token as a case-insensitive substring, so GPTBot and gptbot are the same rule. Exact spelling of the token still matters.
02
It is an honor system
A rule only works if the crawler chooses to obey it. Reputable bots from OpenAI, Anthropic, Perplexity and Google do. Others do not.
03
Bytespider ignores robots.txt
ByteDance's Bytespider routinely crawls despite a Disallow. Blocking it here is advisory only. To actually stop it, block at your WAF or CDN.
04
OpenAI takes about 24 hours
OpenAI caches robots.txt and takes roughly 24 hours to honor a change. Edit it, then give it a day before you expect new behavior.
05
Separate bots, separate decisions
Blocking a training bot (GPTBot, ClaudeBot) does not block the search bot (OAI-SearchBot, Claude-SearchBot). You can stay in answers while withholding training data.
Questions about robots.txt for AI.
Should I block AI crawlers?
Usually not the search and citation bots. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from AI answers, and AI-referred traffic tends to convert well. The default stance here allows search and training and blocks only scraper-only bots. If you want to keep your content out of model training while staying citable, use the cautious preset.
What is the difference between a search bot and a training bot?
A search or citation bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetches your page to answer a live user query and can cite you. A training bot (GPTBot, ClaudeBot, Google-Extended) collects content to train a model. They use different tokens, so you control them independently.
Why are CCBot and Bytespider blocked by default?
Both are scraper-only: they give you no search visibility. CCBot feeds Common Crawl, a third-party archive used to train many models. Bytespider is ByteDance's training crawler and routinely ignores robots.txt, so blocking it here is advisory and a WAF is the real control.
Does robots.txt guarantee a bot will stop?
No. robots.txt is an honor system. Reputable crawlers obey it; others ignore it. For bots that ignore it, such as Bytespider, block at your WAF or CDN instead.
How long until changes take effect?
It depends on the crawler's cache. OpenAI takes roughly 24 hours to honor a robots.txt change. Plan for a day before you expect new behavior.
Where do I put the file?
At the root of your domain, reachable at https://yourdomain.com/robots.txt. Download the generated file and deploy it there.