AI crawlers: what each bot feeds, and what blocking it costs

Microsoft announced two controls on 22nd Sep 2023. Both are meta tags read from pages already in Bing's index.

With NOCACHE, "content with the NOCACHE tag may be included in Bing Chat answers. We will only display URL/Snippet/Title". Only those three parts may be used in training.

With NOARCHIVE, content "will not be included in Bing Chat answers, not be linked to in the answers". Microsoft adds that it "will not use the content for training".

Either way, "content with the NOCACHE tag or NOARCHIVE tag will still appear in our search results".

Bing Chat has since become Copilot Search. Our Microsoft Copilot tracker follows what Bing's rules say today.

Google calls Google-Extended "a standalone product token". It manages whether crawled content "may be used for training future generations of Gemini models" and for grounding "in Gemini Apps and Grounding with Google Search on Vertex AI".

It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search".

It runs no crawler of its own: "Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity."

So the token does not touch AI Overviews or AI Mode. They run on normal Search indexing, and blocking Googlebot is the only crawler-level opt-out.

One study complicates the picture without contradicting Google. A SIGIR 2026 paper found sites blocking Google-Extended were cited less often in AI Overviews.

That is an association, not a proven cause. We cover it on the AI Overviews tracker.

OpenAI says of ChatGPT-User that robots.txt rules "may not apply" because a person triggered the visit. Perplexity says Perplexity-User "generally ignores robots.txt rules" for the same reason.

Meta says meta-externalfetcher "may bypass robots.txt rules" when it "fetches individual links at a user's request".

Anthropic and Mistral have user fetchers too, Claude-User and MistralAI-User. Both say those fetchers obey robots.txt.

Anthropic's Claude Duet agent adds a third kind of visit, from a cloud browser. Anthropic said on 23rd Oct that its requests carry "a distinct User-Agent header" and come from published IP addresses.

It has named neither, and its crawler help page did not list Duet on 24th Oct.

The practical difference: a robots.txt block stops training bots and search bots. The three fetchers above stop only at server rules on their published IP ranges.

Cloudflare's August 2025 accusation that Perplexity crawled past no-crawl directives is on the Perplexity tracker. Perplexity disputed it.

Meta's updated Commercial Terms, first due on 30th Oct 2026 and now 20th Nov, add a crawling clause. The current terms of 6th Mar 2024 have none.

Business account holders "authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials". The targets are "the domains, web pages, or URLs" that "are associated with your products or services, or that are made available through your or users' use of Meta Company Products".

The purpose clause changed between two readings. Archived copies from 16th Oct 2026, the last at 16:20 UTC, let Meta use the content "to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products".

By 03:19 UTC on 17th Oct, archived again, the sentence read "to provide, develop, improve, research, and maintain advertising and business products" and "maintain the integrity and security of Meta Company Products". In the US English editions, "train" is gone from both the Commercial Terms and the Self-Serve Ad Terms previews.

Read again on 19th Oct, the two previews differed by language edition. The UK English editions of both still said "research, train and maintain the integrity of".

PPC Land reported the UK text on 18th Oct. Meta did not say which edition binds a business.

Update, 20th Oct: both UK editions now carry the revised clause. They read "research and maintain advertising and business products" and "maintain the integrity and security of Meta Company Products".

"Train" appears in none of the four editions we read that evening, and none mentions robots.txt. Meta has not announced the UK change, and no outlet had reported it by then.

Archived copies of the UK commercial terms and the UK ad terms previews hold the revised text. Both were captured after 15:00 UTC on 20th Oct.

Meta explained the revision to two outlets on 17th Oct, not in a post of its own. Spokesperson Daniel Roberts told The Verge that the consolidated crawling terms "only applied to our ads and business products".

He added: "As always, content owners can use robots.txt to signal their preferences." The updates "don't give us new rights that supersede those preferences".

A Meta spokesperson told Nieman Lab that "it wasn't clear in the initial consolidation of terms that robots.txt preferences are respected". Nieman Lab notes that the revised terms still carry no robots.txt language.

The in-force ad terms of 22nd Aug 2025 already let Meta crawl advertisers' domains for "research and training".

The new clause reaches business accounts that buy no ads. Nieman Lab reported it on 15th Oct, quoting the "train" version in full.

A Meta spokesperson told Nieman Lab the terms do not change how Meta crawls publishers. "Content owners can still manage how their content is accessed by using robots.txt."

Neither preview mentions robots.txt. Search Engine Journal reports that Meta declined to say whether test credentials would bypass paywalls.

Spokesperson Andy Stone wrote on 16th Oct that the update "is designed to make sure we're able to pull accurate information about products for sale on third-party websites when making on-platform ad and content recommendations, not about scraping news sites". Search Engine Journal and PPC Land both carry it; PPC Land's 17th Oct piece does not note the dropped word "train".

The European Publishers Council said on 16th Oct that Meta "appears to be using its commercial terms to claim sweeping permission to scrape publishers' websites". It is writing to Commission Executive Vice-Presidents Ribera and Virkkunen.

The UK News Media Association called the terms a "shameless landgrab attempt" on 16th Oct, per Nieman Lab. The News/Media Alliance's Danielle Coffey called the update "outrageous and probably illegal".

Update, 22nd Oct: the UK Competition and Markets Authority "sent a formal letter to Meta to discuss its new commercial terms of service", Nieman Lab reports. Its digital markets director Will Hayter told a House of Lords committee on 20th Oct that he had scheduled a meeting with Meta.

CMA chief executive Sarah Cardell told the same session the authority has not designated Meta as having "entrenched market power", which limits its options. It is "urgently" seeking more information from Meta.

In Germany, the publisher bodies BDZV and MVFP asked the European Commission on 20th Oct to examine the change at once and to bar its application until that review ends. They argue it breaches Article 6(12) of the Digital Markets Act.

Dropping "training" does not solve the problem, they write: grounding, retrieval-augmented generation, semantic search and summaries "sind weiterhin erfasst", remain covered. The Commission had not responded in anything we read, and the CMA has announced nothing beyond the letter and the meeting.

Update, 22nd Oct, later that day: the News/Media Alliance delivered a letter to Mark Zuckerberg. It asks Meta to withdraw the revised terms and "to respond publicly by October 26, 2026".

The letter says members "do not consent to them and reserve all of their legal rights". It calls Meta's conduct "an exercise of market power" and anticompetitive.

Publishers "cannot practically disallow or block Meta's crawlers without harming their ability to promote content", it argues. The alliance's release narrows the demand to "the crawling provision" and mentions no complaint to a regulator.

Nieman Lab reports that the Movement for an Open Web "filed formal complaints" on 22nd Oct. They went to the CMA and the European Commission's competition directorate.

Press Gazette confirms the filings. The group's own site carried no complaint post when we read it on 23rd Oct.

WAN-IFRA sent "a similar letter" to Meta's Joel Kaplan the same day, per Nieman Lab. Four European press associations wrote jointly to the Commission on 21st Oct, arguing the terms breach the Digital Markets Act.

Nieman Lab names the European Magazine Media Association, the European Newspaper Publishers' Association and News Media Europe among them. It viewed the letter, which is unpublished and asks the Commission to block the terms "in their current wording".

Cardell also told the committee the CMA is "very alive to the potential impact" on UK businesses, Press Gazette reports. The authority cannot act formally until it designates Meta with strategic market status, the paper adds.

Meta had not replied to the letters in anything we read that morning. The dates to watch were Meta's 26th Oct deadline and the 30th Oct effective date.

Update, 23rd Oct, evening: Meta has put the terms back three weeks. The US English previews of both now read "These updates will go into effect on November 20, 2026" and carry that effective date.

The UK English editions lagged again. At 15:18 UTC both still said "These updates will come into effect on 30 October 2026".

Archived copies date the US change to between 20th Oct at 16:01 UTC and 22nd Oct at 23:14 UTC. The earlier copy still said "October 30, 2026", and Meta announced the change in no post we found.

Spokesperson Daniel Roberts told The Verge the delay is "to allow more time for us to explain to our partners and stakeholders what's changing and what's not". The Verge credits Fast Company with the first report.

Roberts said Meta would use the time to lay out "how Meta uses web crawlers and the fact that content owners will still be able to use robots.txt to signal preferences". The crawling clause itself read the same in all four editions that evening, and none mentioned robots.txt.

For site owners one question is now answered in statements, not in the terms: Meta says robots.txt preferences stand. Which agent does the crawling stays open.

The table above lists Meta's documented tokens. The terms name none.

Update, 25th Oct: the European Publishers Council has written again to the Commission, it said on 23rd Oct. Removing "train", it argues, "does not establish an express exclusion of AI training, fine-tuning, grounding, or the generation and display of answers and summaries".

The authorisation "remains extensive", it says, covering indexing, caching, retrieval, "the creation of test credentials and the periodic collection of updates". It adds that "Meta's references to robots.txt do not resolve this concern".

The council asks the Commission to examine the terms under the Digital Markets Act and EU competition law. If interim measures cannot come before 20th Nov, it wants Meta to undertake to postpone "the contested provisions".

The letter itself is unpublished. The post gives no send date and names no Commission reply, as PPC Land noted on 24th Oct.

Update, 26th Oct: the UK English, German and French previews now carry the new date too. Read at 15:05 UTC, each banner says the updates "will come into effect on 20 November 2026", or the German or French equivalent.

The document body of each still says "Effective date: 30 October 2026". The US editions say 20th Nov in both places.

Our morning read at about 03:00 UTC still showed 30th Oct on the UK editions. So the banners moved during the day.

An archived copy from 15:09 UTC shows both dates.

The purpose clause is unchanged in all six editions. None names robots.txt.

Regulators have spoken in statements, not orders. A Commission spokesperson told Nieman Lab on 23rd Oct that it "welcomes Meta's announcement that it has postponed the entry into force of its new commercial terms".

The Commission is "committed to ensuring that Meta's practices comply in full with all applicable obligations under EU law, including Article 102 TFEU and the Digital Markets Act".

A CMA spokesperson said the same day: "The CMA has been in urgent discussions with Meta this week on this important issue." It "will continue to scrutinize these terms and engage with Meta and others as we consider our next steps".

Theo Bamber of the UK News Media Association wrote that "this delay is welcome, but it is a pause not a solution". Nieman Lab adds that as of 23rd Oct no version of the terms includes "language about respecting robots.txt preferences".

Search Engine Journal reported on 26th Oct that WAN-IFRA's Stig Ørskov called the postponement "a welcome step" that "does not address our concerns". The News/Media Alliance had asked Meta to reply by 26th Oct; no reply was public that evening.

OpenAI is clear for search: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Blocking GPTBot alone leaves search untouched.

Google is clear for training: Google-Extended costs nothing in Search. Apple and DuckDuckGo say the same of Applebot-Extended and DuckAssistBot.

No company says what blocking its training bot costs in visits. One outside study does.

A Wharton and Rutgers working paper studied the 2023 blocking wave. It put the cost at about 7% of weekly traffic, and it is on the Claude tracker.

That study predates AI Overviews and pools every token. Treat it as a reason to measure your own referrals, not as a forecast.

HasData, measured 16th Sep 2026. The scraping-API vendor re-measured 10,894 domains the day after Cloudflare's 15th Sep managed robots.txt deadline.

Its sample includes 1,148 news publishers, and its business depends on crawlers getting through.

The share of sites announcing a block in robots.txt "fell from 15.2% to 13.0%". Among sites behind Cloudflare, the share disallowing GPTBot fell "from 17.1% to 9.9%"; sites not behind Cloudflare moved from 18.7% to 18.6%.

Live blocking went the other way. In its 2,096-site test, 125 sites started blocking GPTBot and 66 stopped, and "112 of the 125 are news publishers".

Sites that block GPTBot without saying so in robots.txt "grew from 115 to 178". The page carries no publication date; PPC Land reported the re-run on 8th Oct.

Nobody measures AI bot traffic across the whole web. Three panels published figures in 2026, and they count different things, so read each on its own terms.

51Degrees, year to 29th May 2026. The data company analysed about three billion visits to sites it works with, across telecoms, advertising, media, retail, finance and technology, as Press Gazette reported on 3rd Sep.

AI bots were 13% of web sessions in May 2026, up from 8% a year earlier, a 62.5% rise in AI-bot traffic over the year. AmazonBot alone was over 4% of all visits and Meta-External-Agent almost 3%.

Those two were 63% of the sessions from the 72 AI-related bots 51Degrees identified. ClaudeBot ranked seventh and OpenAI's search bot ninth; the main Googlebot was outside the study.

DataDome, April to June 2026. The bot-management company's quarterly report counted 17.7 billion AI agent requests across more than 400 enterprise customers, up 45% on the first quarter.

Meta's two agents made 9.1 billion of them, "more than half of all AI traffic on the network", and "sent almost no real visitors back". Meta-WebIndexer, the answer-side crawler, overtook the training-side Meta-ExternalAgent in June.

ChatGPT ran the other way. Its live-fetch agent made 6% fewer requests than in the first quarter, while ChatGPT referrals to websites grew 17% and were "upwards of 80%" of all AI referral traffic.

DataDome also says 54% of its customers had adopted an "agent trust policy", a written rule set for which automated clients may reach which parts of a site.

News UK, six months to September 2026. Its chief technology officer Tom Jackson told Press Gazette's conference that about a fifth of the publisher's traffic was non-human, and that about a third of that was blocked as "malicious bot traffic", as reported on 23rd Sep.

The three do not add up to one number. 51Degrees counts sessions on mixed-industry sites, DataDome counts requests on enterprise sites, and News UK is one publisher's own logs.

Wikimedia Foundation, 5th Oct 2026. Its investigation found agents it believes OpenAI operates made "millions of automated requests" to its public APIs and crawled millions of pages, mainly on Wikidata and Wikimedia Commons.

They also made "hundreds of thousands" of Wikidata Query Service queries, which "may have contributed to a partial outage" in May. The attribution is Wikimedia's belief, not an OpenAI confirmation, and the post gives no share of total traffic.

The same post reports sandbox test edits and failed attempts to use a hosted note-taking tool as a proxy. Wikimedia found no evidence its systems or data were compromised.

The policy response is under way. More than 300 news executives went to Washington on 29th Sep for the Stealth Bot Prohibition Act, which would require crawlers to identify themselves and disclose their purpose, as Search Engine Land reported.

Press Gazette's piece also carries a robots.txt figure from the analytics firm Known Agents: publishers block about 21% of AI scraper bots in their files. The table above is the list to check yours against.

OpenAI publishes ranges for each bot at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. Anthropic publishes claude.com/crawling/bots.json.

Perplexity publishes perplexity.com/perplexitybot.json and perplexity-user.json. DuckDuckGo publishes duckduckgo.com/duckassistbot.json, and Amazon publishes its list at developer.amazon.com/amazonbot/ip-addresses.

Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot" and recommends reverse DNS checks.

ByteDance publishes no IP list for Bytespider that we could find. A request carrying that string cannot be checked either way.

# robots.txt: the training crawlers documented above
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
Disallow: /

That block leaves every search crawler alone. Pages stay eligible for ChatGPT search, Perplexity, Claude search, Meta AI search and DuckAssist.

Bing needs the meta tags instead.

Cloudflare said on 16th Oct 2026 that it now measures "how often a crawler respects site preferences" for its Verified bots. The first signals are "Does the bot respect instructions?" and "Did it respond to direct feedback?"

The observations become "an explainable trust band" per bot. Cloudflare will share the scores "publicly on Radar".

It plans to let customers "choose the trust threshold". It names no scores and no date.