Every AI company that answers questions from the web runs bots. They are not all the same bot, and blocking the wrong one costs you reach for nothing.
This page lists every documented token in one table, with each company's own words for what it feeds. We read each company's page on 6th Sep 2026.
Three kinds of bot. A training crawler collects pages to train models.
A search crawler indexes pages so an assistant can cite them.
A user fetcher opens one page because a person asked. Several companies say it may ignore robots.txt.
Every documented AI crawler and control
Token, operator, what it feeds, robots.txt behaviour and the cost of blocking, in each company's own words.
| Token | Operator | Kind | What the company says it feeds | Obeys robots.txt? | What blocking it costs |
|---|---|---|---|---|---|
GPTBot | OpenAI | Training | Used "to make our generative AI foundation models more useful and safe" | Yes | Nothing in ChatGPT search; OpenAI says disallowing it "indicates a site's content should not be used in training" |
OAI-SearchBot | OpenAI | Search | "Used to surface websites in search results in ChatGPT's search features" | Yes | "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers" |
ChatGPT-User | OpenAI | User fetcher | "For certain user actions in ChatGPT and Custom GPTs" | OpenAI says robots.txt rules "may not apply" to these visits | Not stated |
OAI-AdsBot | OpenAI | Ads check | Validates "the safety of web pages submitted as ads on ChatGPT" | Not stated | Not stated; affects advertisers, not publishers |
ClaudeBot | Anthropic | Training | Collects "web content that could potentially contribute to their training" | Yes, and honours Crawl-delay | Not stated |
Claude-SearchBot | Anthropic | Search | "Navigates the web to improve search result quality for users" | Yes | Not stated |
Claude-User | Anthropic | User fetcher | "When individuals ask questions to Claude, it may access websites using a Claude-User agent" | Yes, per Anthropic | Not stated |
PerplexityBot | Perplexity | Search | "Designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models" | Yes | Perplexity recommends allowing it so pages can be surfaced |
Perplexity-User | Perplexity | User fetcher | "Supports user actions within Perplexity" | "Generally ignores robots.txt rules", per Perplexity | Not stated |
Google-Extended | Control, not a crawler | Manages whether crawled content trains Gemini models and grounds answers in Gemini Apps and Vertex AI | It is a robots.txt token; Google says it has "no separate HTTP request user agent string" | "Does not impact a site's inclusion in Google Search nor is it used as a ranking signal" | |
Google-CloudVertexBot | Enterprise | Crawls sites for Vertex AI Agents at the site owner's request | Yes | Affects only the owner's own agents | |
Applebot | Apple | Search | Powers "Spotlight, Siri, and Safari" search features | Yes | Removes pages from those features |
Applebot-Extended | Apple | Control, not a crawler | Governs whether crawled data helps "train Apple foundation models"; "Applebot-Extended does not crawl webpages" | It is a robots.txt token | None in Apple search features |
meta-externalagent | Meta | Training and indexing | "Training foundation AI models or improving products by indexing content directly" | Yes | Not stated |
meta-webindexer | Meta | Search | "Navigates the web to improve Meta AI search result quality" | Yes | Not stated |
meta-externalfetcher | Meta | User fetcher | "Fetches individual links at a user's request" | "May bypass robots.txt rules", per Meta | Not stated |
Amazonbot | Amazon | Products and training | "Used to improve our products and services" and "may be used to train Amazon AI models" | Yes; caches robots.txt up to 30 days | Not stated |
DuckAssistBot | DuckDuckGo | Search | "Crawls pages in real-time for our AI-assisted answers, which prominently cite their sources" | Yes; changes apply "after 72 hours" | "Opting out of DuckAssistBot does not impact organic search rankings" |
MistralAI-Training | Mistral | Training | "Crawls web content to help build datasets for training Mistral generative AI models" | Yes | Not stated |
MistralAI-Index | Mistral | Search | "Automated crawling of the web for indexing purposes only" | Yes | Not stated |
MistralAI-User | Mistral | User fetcher | "For user actions in Vibe" | Yes, per Mistral | Not stated |
CCBot | Common Crawl | Open archive | An "open repository of web crawl data"; widely used as training data by others | Yes | Not stated; Common Crawl warns of bots "falsely identifying themselves as CCBot" |
Bytespider | ByteDance | Undocumented | No operator documentation found; the user-agent string points to a Toutiao webmaster page | Not stated by ByteDance | Not stated |
| No token | xAI (Grok) | Undocumented | xAI publishes no crawler token, so publishers have no robots.txt control over Grok | Not applicable | Not applicable |
Microsoft is the exception. Bing indexes with Bingbot, and its AI controls are meta tags, covered in the next section.
Bing and Copilot: tags, not tokens
Microsoft lets pages stay in Bing search while opting out of Copilot answers and model training.
Microsoft announced two controls on 22nd Sep 2023. Both are meta tags read from pages already in Bing's index.
With NOCACHE, "content with the NOCACHE tag may be included in Bing Chat answers. We will only display URL/Snippet/Title". Only those three parts may be used in training.
With NOARCHIVE, content "will not be included in Bing Chat answers, not be linked to in the answers". Microsoft adds that it "will not use the content for training".
Either way, "content with the NOCACHE tag or NOARCHIVE tag will still appear in our search results".
Bing Chat has since become Copilot Search. Our Microsoft Copilot tracker follows what Bing's rules say today.
Google-Extended: what it does and does not gate
It gates Gemini training and grounding. It gates nothing in Search, AI Overviews or AI Mode.
Google calls Google-Extended "a standalone product token". It manages whether crawled content "may be used for training future generations of Gemini models" and for grounding "in Gemini Apps and Grounding with Google Search on Vertex AI".
It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search".
It runs no crawler of its own: "Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity."
So the token does not touch AI Overviews or AI Mode. They run on normal Search indexing, and blocking Googlebot is the only crawler-level opt-out.
One study complicates the picture without contradicting Google. A SIGIR 2026 paper found sites blocking Google-Extended were cited less often in AI Overviews.
That is an association, not a proven cause. We cover it on the AI Overviews tracker.
The fetchers that say they may ignore robots.txt
Three companies document a user-triggered fetch that robots.txt may not stop.
OpenAI says of ChatGPT-User that robots.txt rules "may not apply" because a person triggered the visit. Perplexity says Perplexity-User "generally ignores robots.txt rules" for the same reason.
Meta says meta-externalfetcher "may bypass robots.txt rules" when it "fetches individual links at a user's request".
Anthropic and Mistral have user fetchers too, Claude-User and MistralAI-User. Both say those fetchers obey robots.txt.
The practical difference: a robots.txt block stops training bots and search bots. The three fetchers above stop only at server rules on their published IP ranges.
Cloudflare's August 2025 accusation that Perplexity crawled past no-crawl directives is on the Perplexity tracker. Perplexity disputed it.
What blocking actually costs
Only three companies state a cost. The rest leave publishers to measure it.
OpenAI is clear for search: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Blocking GPTBot alone leaves search untouched.
Google is clear for training: Google-Extended costs nothing in Search. Apple and DuckDuckGo say the same of Applebot-Extended and DuckAssistBot.
No company says what blocking its training bot costs in visits. One outside study does.
A Wharton and Rutgers working paper studied the 2023 blocking wave. It put the cost at about 7% of weekly traffic, and it is on the Claude tracker.
That study predates AI Overviews and pools every token. Treat it as a reason to measure your own referrals, not as a forecast.
Verifying a bot is who it says it is
Most companies publish IP ranges as JSON files. Anyone can type a user-agent string.
OpenAI publishes ranges for each bot at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. Anthropic publishes claude.com/crawling/bots.json.
Perplexity publishes perplexity.com/perplexitybot.json and perplexity-user.json. DuckDuckGo publishes duckduckgo.com/duckassistbot.json, and Amazon publishes its list at developer.amazon.com/amazonbot/ip-addresses.
Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot" and recommends reverse DNS checks.
ByteDance publishes no IP list for Bytespider that we could find. A request carrying that string cannot be checked either way.
# robots.txt: the training crawlers documented above
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
Disallow: /That block leaves every search crawler alone. Pages stay eligible for ChatGPT search, Perplexity, Claude search, Meta AI search and DuckAssist.
Bing needs the meta tags instead.
Sources
- OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot)
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers (PerplexityBot, Perplexity-User)
- Google: Google's common crawlers, including the Google-Extended token
- Apple: About Applebot and Applebot-Extended
- Meta: Meta web crawlers
- Amazon: Amazonbot
- DuckDuckGo: DuckAssistBot
- Mistral: Mistral AI crawlers
- Common Crawl: CCBot
- Microsoft Bing: Announcing new options for webmasters to control usage of their content in Bing Chat (22nd Sep 2023)
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No, according to OpenAI. GPTBot feeds training, and OAI-SearchBot builds the search index.
Only opting out of OAI-SearchBot removes a site from ChatGPT search answers.
Does blocking Google-Extended affect AI Overviews or AI Mode?
Not according to Google. The token controls Gemini training and grounding, and Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".
A SIGIR 2026 study found sites that block it were cited less in AI Overviews. We report that study on the AI Overviews tracker.
Which AI bots ignore robots.txt?
Three operators say their user-triggered fetchers may: OpenAI's ChatGPT-User, Perplexity's Perplexity-User and Meta's meta-externalfetcher.
Every training and search crawler in the table says it obeys robots.txt. ByteDance publishes no statement about Bytespider.
How do I block AI training without leaving AI search?
Disallow the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, MistralAI-Training, CCBot) and leave the search tokens alone. For Bing, add the NOARCHIVE meta tag to opt out of Copilot answers and training while staying in search results.
Is there a Grok crawler?
xAI publishes no crawler token, so publishers have no robots.txt control over Grok. Our Grok tracker records how that was checked and will note any change.
Compiled by Claude from primary sources, checked by Akash Mansukhani before publication. See how we source and correct this page.Keywords Everywhere, "AI crawlers: what each bot feeds, and what blocking it costs", last updated 6th Sep 2026, https://keywordseverywhere.com/news/ai-crawlers/