AI crawlers: what each bot feeds, and what blocking it costs

Microsoft announced two controls on 22nd Sep 2023. Both are meta tags read from pages already in Bing's index.

With NOCACHE, "content with the NOCACHE tag may be included in Bing Chat answers. We will only display URL/Snippet/Title". Only those three parts may be used in training.

With NOARCHIVE, content "will not be included in Bing Chat answers, not be linked to in the answers". Microsoft adds that it "will not use the content for training".

Either way, "content with the NOCACHE tag or NOARCHIVE tag will still appear in our search results".

Update, 30th Oct 2026: Microsoft announced replacements. From 1st Feb 2027 Bing reads the shared training token in robots.txt.

It also reads page-level ai-training and ai-grounding tags addressed to Bingbot.

Grounding gets three settings: Allow, DisallowExtended (Microsoft's own products only, not its third-party grounding services) and Disallow. After 1st Mar 2027 "NoArchive and NoCache will no longer be read as a preference for training or grounding".

Update, 31st Oct 2026: Bing's help FAQ, undated, adds a default. Pages using the old tags with no temporary preference recorded by 1st Mar 2027 "default to Allow for grounding until Microsoft recrawls them".

Our Copilot tracker entry has the dates and the transition steps.

Bing Chat has since become Copilot Search. Our Microsoft Copilot tracker follows what Bing's rules say today.

Google calls Google-Extended "a standalone product token". It manages whether crawled content "may be used for training future generations of Gemini models" and for grounding "in Gemini Apps and Grounding with Google Search on Vertex AI".

It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search".

It runs no crawler of its own: "Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity."

So the token does not touch AI Overviews or AI Mode. They run on normal Search indexing, and blocking Googlebot is the only crawler-level opt-out.

On 30th Oct 2026 Google announced a second way to disallow it. John Mueller's post says Google plans "to support a shared AI training control in robots.txt".

The control is a "policy token" written in the User-agent line. The crawler documentation names it: __usage_ai-training.

When a site opts out of training with that token "and Google-Extended is not explicitly specified", Google "plans to interpret that as if the publisher had disallowed Google-Extended". A site that names Google-Extended keeps its own rule.

Google says it aims to support the control "along with other AI providers in 2027". The common crawlers page, stamped 30th Oct, says "in early 2027".

The scope is the same as Google-Extended's. The post says nothing about Googlebot, AI Overviews, AI Mode or ranking.

Microsoft and OpenAI announced the same token the same day. The next section compares the four companies' words.

One study complicates the picture without contradicting Google. A SIGIR 2026 paper found sites blocking Google-Extended were cited less often in AI Overviews.

That is an association, not a proven cause. We cover it on the AI Overviews tracker.

Google, Microsoft and OpenAI announced support in their own posts on 30th Oct 2026. Search Engine Roundtable and The Verge name Anthropic as a fourth; we found no Anthropic post.

Update, 5th Nov: Anthropic's crawler help page, edited 4th Nov, now says that "As of October 30, 2026, Anthropic has begun recognizing" the signal. There is still no Anthropic post.

The signal is an individual Internet-Draft by Kevin Kelley, dated 30th Oct. It binds a preference to paths with "ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line".

The token is __usage_ai-training. The draft "introduces no new directive names and no new grammar", and the IETF page says it "has no formal standing in the IETF standards process".

# the draft's own example
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Google: a site that opts out with the token "and Google-Extended is not explicitly specified" is read "as if the publisher had disallowed Google-Extended". The blog says "in 2027" and the docs "in early 2027".

Google-Extended gates Gemini training and grounding, not Search. Google told Search Engine Roundtable it is "moving forward with this initiative alongside peer AI providers".

Microsoft: Bing will honour the token from 1st Feb 2027, with page-level tags that can override it. The same post adds grounding tags and retires NoArchive and NoCache; see Bing and Copilot.

OpenAI: it "will honor a common signal" and aims "to implement the signal in the coming months". Its post names no token and says the existing crawler rules "will continue to apply and be respected".

Anthropic: The Verge reports that Kelley is "a technical lead for standards at Anthropic" and that Anthropic "has already adopted the signal". Anthropic's crawler help page, read on 31st Oct, does not mention the token.

Update, 5th Nov: the help page, edited 4th Nov, says Anthropic will "honor a new preference signal" that lets site owners say "no AI training". It is "rolling this out across all of our services over time and globally".

A robots.txt rule naming ClaudeBot takes priority: "If your robots.txt file names ClaudeBot, we follow that rule, even if your general setting says something different." The page links the IETF draft but never prints the token. Our Claude tracker has the entry.

Meta made no commitment. Its spokesperson Andy Stone told The Verge that a common signal across providers "is an important step".

What it does not cover: grounding, search answers or citations. Every provider's search and fetcher tokens stay as they are, and nobody has given the day the token takes effect.

OpenAI says of ChatGPT-User that robots.txt rules "may not apply" because a person triggered the visit. Perplexity says Perplexity-User "generally ignores robots.txt rules" for the same reason.

Meta says meta-externalfetcher "may bypass robots.txt rules" when it "fetches individual links at a user's request".

Anthropic and Mistral have user fetchers too, Claude-User and MistralAI-User. Both say those fetchers obey robots.txt.

Anthropic's Claude Duet agent adds a third kind of visit, from a cloud browser. Anthropic said on 23rd Oct that its requests carry "a distinct User-Agent header" and come from published IP addresses.

It has named neither, and its crawler help page did not list Duet on 24th Oct.

Update, 5th Nov: a new Anthropic help article names both. The browser's User-Agent carries the token Claude-Agent, and its IP addresses are published at claude.com/.well-known/sail.

The article says preferences for Anthropic's crawlers "are independent" of this traffic. It gives no robots.txt token for the browser and does not say whether the browser reads robots.txt.

Google's own fetches raise a question. Dave Smart of Tame the Bots wrote on 3rd Nov about two fetches in his logs.

A Gemini fetch arrived as the bare "Google" user-agent, an AI Mode fetch as "Google-Lens". That is one practitioner's log observation, "subject to change" in his own words.

Google's user-triggered fetchers page, last updated 19th Aug, lists neither string.

The practical difference: a robots.txt block stops training bots and search bots. The three fetchers above stop only at server rules on their published IP ranges.

Cloudflare's August 2025 accusation that Perplexity crawled past no-crawl directives is on the Perplexity tracker. Perplexity disputed it.

Meta's updated Commercial Terms, first due on 30th Oct 2026, then 20th Nov, and paused on 5th Nov, add a crawling clause. The current terms of 6th Mar 2024 have none.

Business account holders "authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials". The targets are "the domains, web pages, or URLs" that "are associated with your products or services, or that are made available through your or users' use of Meta Company Products".

The purpose clause changed between two readings. Archived copies from 16th Oct 2026, the last at 16:20 UTC, let Meta use the content "to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products".

By 03:19 UTC on 17th Oct, archived again, the sentence read "to provide, develop, improve, research, and maintain advertising and business products" and "maintain the integrity and security of Meta Company Products". In the US English editions, "train" is gone from both the Commercial Terms and the Self-Serve Ad Terms previews.

Read again on 19th Oct, the two previews differed by language edition. The UK English editions of both still said "research, train and maintain the integrity of".

PPC Land reported the UK text on 18th Oct. Meta did not say which edition binds a business.

Update, 20th Oct: both UK editions now carry the revised clause. They read "research and maintain advertising and business products" and "maintain the integrity and security of Meta Company Products".

"Train" appears in none of the four editions we read that evening, and none mentions robots.txt. Meta has not announced the UK change, and no outlet had reported it by then.

Archived copies of the UK commercial terms and the UK ad terms previews hold the revised text. Both were captured after 15:00 UTC on 20th Oct.

Meta explained the revision to two outlets on 17th Oct, not in a post of its own. Spokesperson Daniel Roberts told The Verge that the consolidated crawling terms "only applied to our ads and business products".

He added: "As always, content owners can use robots.txt to signal their preferences." The updates "don't give us new rights that supersede those preferences".

A Meta spokesperson told Nieman Lab that "it wasn't clear in the initial consolidation of terms that robots.txt preferences are respected". Nieman Lab notes that the revised terms still carry no robots.txt language.

The in-force ad terms of 22nd Aug 2025 already let Meta crawl advertisers' domains for "research and training".

The new clause reaches business accounts that buy no ads. Nieman Lab reported it on 15th Oct, quoting the "train" version in full.

A Meta spokesperson told Nieman Lab the terms do not change how Meta crawls publishers. "Content owners can still manage how their content is accessed by using robots.txt."

Neither preview mentions robots.txt. Search Engine Journal reports that Meta declined to say whether test credentials would bypass paywalls.

Spokesperson Andy Stone wrote on 16th Oct that the update "is designed to make sure we're able to pull accurate information about products for sale on third-party websites when making on-platform ad and content recommendations, not about scraping news sites". Search Engine Journal and PPC Land both carry it; PPC Land's 17th Oct piece does not note the dropped word "train".

The European Publishers Council said on 16th Oct that Meta "appears to be using its commercial terms to claim sweeping permission to scrape publishers' websites". It is writing to Commission Executive Vice-Presidents Ribera and Virkkunen.

The UK News Media Association called the terms a "shameless landgrab attempt" on 16th Oct, per Nieman Lab. The News/Media Alliance's Danielle Coffey called the update "outrageous and probably illegal".

Update, 22nd Oct: the UK Competition and Markets Authority "sent a formal letter to Meta to discuss its new commercial terms of service", Nieman Lab reports. Its digital markets director Will Hayter told a House of Lords committee on 20th Oct that he had scheduled a meeting with Meta.

CMA chief executive Sarah Cardell told the same session the authority has not designated Meta as having "entrenched market power", which limits its options. It is "urgently" seeking more information from Meta.

In Germany, the publisher bodies BDZV and MVFP asked the European Commission on 20th Oct to examine the change at once and to bar its application until that review ends. They argue it breaches Article 6(12) of the Digital Markets Act.

Dropping "training" does not solve the problem, they write: grounding, retrieval-augmented generation, semantic search and summaries "sind weiterhin erfasst", remain covered. The Commission had not responded in anything we read, and the CMA had announced nothing beyond its letter and meeting.

Update, 22nd Oct, later that day: the News/Media Alliance delivered a letter to Mark Zuckerberg. It asks Meta to withdraw the revised terms and "to respond publicly by October 26, 2026".

The letter says members "do not consent to them and reserve all of their legal rights". It calls Meta's conduct "an exercise of market power" and anticompetitive.

Publishers "cannot practically disallow or block Meta's crawlers without harming their ability to promote content", it argues. The alliance's release narrows the demand to "the crawling provision" and mentions no complaint to a regulator.

Nieman Lab reports that the Movement for an Open Web "filed formal complaints" on 22nd Oct. They went to the CMA and the European Commission's competition directorate.

Press Gazette confirms the filings. The group's own site carried no complaint post when we read it on 23rd Oct.

WAN-IFRA sent "a similar letter" to Meta's Joel Kaplan the same day, per Nieman Lab. Four European press associations wrote jointly to the Commission on 21st Oct, arguing the terms breach the Digital Markets Act.

Nieman Lab names the European Magazine Media Association, the European Newspaper Publishers' Association and News Media Europe among them. It viewed the letter, which is unpublished and asks the Commission to block the terms "in their current wording".

Cardell also told the committee the CMA is "very alive to the potential impact" on UK businesses, Press Gazette reports. The paper added it could not act formally before designating Meta, which the 30th Oct investigation sidestepped.

Meta had not replied to the letters in anything we read that morning. The dates to watch were Meta's 26th Oct deadline and the 30th Oct effective date.

Update, 23rd Oct, evening: Meta has put the terms back three weeks. The US English previews of both now read "These updates will go into effect on November 20, 2026" and carry that effective date.

The UK English editions lagged again. At 15:18 UTC both still said "These updates will come into effect on 30 October 2026".

Archived copies date the US change to between 20th Oct at 16:01 UTC and 22nd Oct at 23:14 UTC. The earlier copy still said "October 30, 2026", and Meta announced the change in no post we found.

Spokesperson Daniel Roberts told The Verge the delay is "to allow more time for us to explain to our partners and stakeholders what's changing and what's not". The Verge credits Fast Company with the first report.

Roberts said Meta would use the time to lay out "how Meta uses web crawlers and the fact that content owners will still be able to use robots.txt to signal preferences". The crawling clause itself read the same in all four editions that evening, and none mentioned robots.txt.

For site owners one question is now answered in statements, not in the terms: Meta says robots.txt preferences stand. Which agent does the crawling stays open.

The table above lists Meta's documented tokens. The terms name none.

Update, 25th Oct: the European Publishers Council has written again to the Commission, it said on 23rd Oct. Removing "train", it argues, "does not establish an express exclusion of AI training, fine-tuning, grounding, or the generation and display of answers and summaries".

The authorisation "remains extensive", it says, covering indexing, caching, retrieval, "the creation of test credentials and the periodic collection of updates". It adds that "Meta's references to robots.txt do not resolve this concern".

The council asks the Commission to examine the terms under the Digital Markets Act and EU competition law. If interim measures cannot come before 20th Nov, it wants Meta to undertake to postpone "the contested provisions".

The letter itself is unpublished. The post gives no send date and names no Commission reply, as PPC Land noted on 24th Oct.

Update, 26th Oct: the UK English, German and French previews now carry the new date too. Read at 15:05 UTC, each banner says the updates "will come into effect on 20 November 2026", or the German or French equivalent.

The document body of each still said "Effective date: 30 October 2026" that evening. The US editions said 20th Nov in both places.

Our morning read at about 03:00 UTC still showed 30th Oct on the UK editions. So the banners moved during the day.

An archived copy from 15:09 UTC shows both dates.

Update, 28th Oct: the document bodies have caught up. Read at about 03:05 UTC, the UK English preview says "Effective from 20 November 2026".

The German edition says "Gültig ab 20. November 2026" and the French "Date effective : 20 novembre 2026". The UK self-service ads preview moved the same way.

So every edition we read now says 20th Nov in both places. The change happened between our reads of 27th and 28th Oct; no archived copy bounds it tighter.

The purpose clause is unchanged in all six editions. None names robots.txt.

Regulators have spoken in statements, not orders. A Commission spokesperson told Nieman Lab on 23rd Oct that it "welcomes Meta's announcement that it has postponed the entry into force of its new commercial terms".

The Commission is "committed to ensuring that Meta's practices comply in full with all applicable obligations under EU law, including Article 102 TFEU and the Digital Markets Act".

A CMA spokesperson said the same day: "The CMA has been in urgent discussions with Meta this week on this important issue." It "will continue to scrutinize these terms and engage with Meta and others as we consider our next steps".

Theo Bamber of the UK News Media Association wrote that "this delay is welcome, but it is a pause not a solution". Nieman Lab adds that as of 23rd Oct no version of the terms includes "language about respecting robots.txt preferences".

Search Engine Journal reported on 26th Oct that WAN-IFRA's Stig Ørskov called the postponement "a welcome step" that "does not address our concerns". The News/Media Alliance had asked Meta to reply by 26th Oct; no reply was public that evening.

Update, 27th Oct: Norway's publishers have gone to court. MBL, the Norwegian publishers' association, says it has asked the Oslo District Court for a temporary injunction.

Its post names NRK, TV 2, Schibsted, Amedia, Polaris, Aller, DN, Mentor Medier, NTB and Hordaland Folkeblad as the applicants. In Norwegian: "Vi har sendt inn krav om midlertidig forføyning", a request for a temporary injunction, filed before any court has ruled on the terms.

MBL sets three demands. Meta must write respect for machine-readable reservations, "blant annet gjennom robots.txt" (among them robots.txt), into the contract terms; AI use must need a separate voluntary agreement; and a refusal must not cost reach or traffic.

No case number or filing date is given, and the court has not ruled. Nieman Lab calls it "the first known lawsuit filed against Meta over its commercial terms".

Press Gazette adds that Danish publishers and the Danish press collecting society have jointly written to their competition authority. Swedish publisher associations "have reported Meta to their competition watchdog".

A CMA spokesperson told Press Gazette on 27th Oct: "The CMA has been in urgent discussions with Meta on this important issue. Meta has now paused implementation until 20 November." It "will continue to scrutinise these terms".

In the US, Senator Adam Schiff wrote to Mark Zuckerberg on 27th Oct, Nieman Lab reports. He wants a reply by 2nd Nov, and asks Meta to "provide explicit guarantees that external technical boundaries like paywalls and robots.txt protocols will be fully respected".

Schiff writes that Meta's "verbal assurances" on robots.txt "do not appear to constitute enforceable legal protections". The letter itself is not published.

Nieman Lab says Senator Marsha Blackburn called on 23rd Oct for the terms to be withdrawn.

Update, 29th Oct: two things happened on 28th Oct. Meta pledged a consultation, and a Norwegian court ordered the terms held.

Andy Stone wrote on X at 15:47 UTC that Meta does "not plan to make policy changes without a full consultation with industry groups and leaders". The consultation "could extend beyond November 20, in which case we'd delay implementation of the updated terms".

"Until then our existing terms remain in place." The pledge is conditional, and Meta named no groups and no new date.

Read at 03:05 UTC on 29th Oct, all five preview editions still say 20th Nov, and "train" is still absent. Engadget notes that Meta "still lists November 20 on its website".

Nieman Lab says Meta "did not answer" which groups it plans to consult. It repeats that "No version of the terms has included language about respecting robots.txt preferences".

Senator Blackburn told Nieman Lab "a vague promise to consult with industry stakeholders and potentially delay implementation is not nearly enough". The EPC's Angela Mills Wade said the consultation "must lead to binding contractual safeguards".

In Oslo, MBL says the District Court "ruled in favor of the industry" on 28th Oct. "This means that Meta is prohibited from implementing the changes to its commercial terms until the legality of those changes has been finally determined by the courts."

The Norwegian post puts it as "Meta forbys å iverksette endringene", Meta is forbidden from implementing the changes. MBL's chair says the court agreed that "Meta holds a dominant position in the relevant market and that the new terms are unreasonable".

That is the claimant's account. The written decision, its case number and its scope are unpublished.

We do not know whether the order reaches beyond the ten applicants. PPC Land attributes to AFP that the court ruled "on an emergency basis without hearing from Meta".

It attributes the competition-law framing to the Swedish agency TT. Nieman Lab, citing NRK, says the order blocks the terms "from taking effect on November 20".

No Meta response to the order was in the coverage we read on 29th Oct. The 20th Nov date on Meta's pages now has a pledge and a court order against it.

Update, 2nd Nov: the CMA has opened a formal investigation. It announced on 30th Oct an investigation "into whether Meta's commercial terms for businesses using its platforms and services breach competition law".

The authority is "using its Competition Act 1998 (CA98) powers so it can urgently investigate the concerns it has heard". It names the News Media Association and Movement for an Open Web among them.

Its concern is that requiring publishers to accept the terms "could amount to an unlawful imposition of unfair trading conditions for thousands of UK businesses". The case page names Meta Platforms, Facebook UK and Facebook Ireland.

The case runs under Chapter II of the Act, the abuse-of-dominance rule. The timetable gives "October 2026 to January 2027" for "Rapid initial information gathering" and says the dates "may change".

Sarah Cardell said the CMA is "acting swiftly to investigate whether Meta's proposed changes may breach competition law". The release adds: "As this is only the start of the CMA's investigation, no decision has yet been made on whether Meta has broken the law."

The release also records Meta's 28th Oct "pause to the roll out of its commercial terms, pending a consultation with industry groups". Meta's pages still say 20th Nov.

So the earlier line that the CMA could not act before designating Meta no longer holds. It acted under the 1998 Act, not the digital markets regime.

Press Gazette carries the publishers' reactions. We read the release on 2nd Nov, three days after it was published.

Update, 4th Nov: a São Paulo court has suspended the crawl clause for Brazil's main media associations. ANJ, the newspaper association, says Judge Guilherme Madeira Dezem of the 44th Civil Court granted an injunction on 3rd Nov.

The order "suspendeu parte dos Termos Comerciais da Meta, previstos para entrar em vigor em 20 de novembro", suspending part of the terms due on 20th Nov. It covers members of ANJ, ABERT (broadcasters) and ANER (magazines).

Meta must keep the members' accounts, pages and tools running "sem bloqueios, restrições ou punições pela recusa das novas condições", without blocks, restrictions or penalties for refusing the new terms. The fine is "R$ 1 milhão" a day, capped at "R$ 100 milhões".

"A liminar vale até o julgamento definitivo da ação": the injunction holds until final judgment. That is the complainant's account; the court has not published the decision.

PPC Land quotes the decision in translation. It gives the case number 4214874-21.2026.8.26.0100, says the judge ruled "without hearing Meta", and that Meta "has five days to contest the order".

In PPC Land's translation the court calls robots.txt "a mere technical cooperation directive without binding legal force". No Meta statement was in the coverage we read on 4th Nov.

Update, 6th Nov: Meta has paused the terms. A new Business Help Center notice says "These updates are now paused and will not take effect on November 20, 2026."

It continues: "We are continuing to consult with stakeholders and will provide a further update when we have more to share. In the meantime, our existing terms remain in effect."

On crawling it adds: "our crawling practices remain unchanged and content owners can continue to signal their crawling preferences using robots.txt as described in our developer documentation."

The in-force US Commercial Terms and Self-Serve Ad Terms now carry a banner. Each says the preview "updates are on hold" and the existing terms "remain in effect until further notice".

The UK English, German and French editions of the Commercial Terms carried no banner that day. All six preview pages now return "Content Not Found".

Update, 7th Nov: the UK English, German and French editions now carry the banner too. The UK English wording matches the US one; the German says "Diese Aktualisierungen werden vorübergehend ausgesetzt" and the French "Ces mises à jour sont en suspens".

None of the three had it at 15:08 UTC on 6th Nov. All three did by 03:11 UTC on 7th Nov, when archived copies were taken.

No new date, text or reason has appeared.

The notice is undated. Our read at 15:10 UTC on 5th Nov still showed the previews saying 20th Nov.

404 Media quoted the notice at 21:01 UTC the same day.

The notice gives no reason, no new date and no revised text. It does not mention the CMA investigation, the Oslo order or the São Paulo order.

No Meta newsroom post or Andy Stone statement accompanied it in anything we read on 6th Nov. The in-force ad terms, which already grant "research and training", are unchanged.

OpenAI is clear for search: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Blocking GPTBot alone leaves search untouched.

Google is clear for training: Google-Extended costs nothing in Search. Apple and DuckDuckGo say the same of Applebot-Extended and DuckAssistBot.

No company says what blocking its training bot costs in visits. One outside study does.

A Wharton and Rutgers working paper studied the 2023 blocking wave. It put the cost at about 7% of weekly traffic, and it is on the Claude tracker.

That study predates AI Overviews and pools every token. Treat it as a reason to measure your own referrals, not as a forecast.

HasData, measured 16th Sep 2026. The scraping-API vendor re-measured 10,894 domains the day after Cloudflare's 15th Sep managed robots.txt deadline.

Its sample includes 1,148 news publishers, and its business depends on crawlers getting through.

The share of sites announcing a block in robots.txt "fell from 15.2% to 13.0%". Among sites behind Cloudflare, the share disallowing GPTBot fell "from 17.1% to 9.9%"; sites not behind Cloudflare moved from 18.7% to 18.6%.

Live blocking went the other way. In its 2,096-site test, 125 sites started blocking GPTBot and 66 stopped, and "112 of the 125 are news publishers".

Sites that block GPTBot without saying so in robots.txt "grew from 115 to 178". The page carries no publication date; PPC Land reported the re-run on 8th Oct.

Seonix, crawled 3rd Nov 2026. The SEO vendor read the robots.txt of 14,318 sites on one day.

Its sample is "DuckDuckGo top 10s for commercial searches chosen for 23 websites that use Seonix".

It measures declarations, not traffic. In its words, a robots.txt "does not show what a bot actually did", and firewall blocks are invisible to it.

12.2% of sites "block at least one AI training bot by name". Only 2.3% "block a bot behind AI search or answers", and 35.8% of the training blocks sit in files carrying Cloudflare's managed template.

"1,414 sites (9.9%) block training bots by name and leave every search and answer bot open. Fewer than 30 do the reverse."

By token, GPTBot is blocked on 8.0% of sites and OAI-SearchBot on 0.7%. ClaudeBot is at 7.4%, Claude-SearchBot at 0.5% and Google-Extended at 6.4%.

The page names no author. It says a single crawl "shows no trend".

Nobody measures AI bot traffic across the whole web. Three panels published figures in 2026, and they count different things, so read each on its own terms.

51Degrees, year to 29th May 2026. The data company analysed about three billion visits to sites it works with, across telecoms, advertising, media, retail, finance and technology, as Press Gazette reported on 3rd Sep.

AI bots were 13% of web sessions in May 2026, up from 8% a year earlier, a 62.5% rise in AI-bot traffic over the year. AmazonBot alone was over 4% of all visits and Meta-External-Agent almost 3%.

Those two were 63% of the sessions from the 72 AI-related bots 51Degrees identified. ClaudeBot ranked seventh and OpenAI's search bot ninth; the main Googlebot was outside the study.

DataDome, April to June 2026. The bot-management company's quarterly report counted 17.7 billion AI agent requests across more than 400 enterprise customers, up 45% on the first quarter.

Meta's two agents made 9.1 billion of them, "more than half of all AI traffic on the network", and "sent almost no real visitors back". Meta-WebIndexer, the answer-side crawler, overtook the training-side Meta-ExternalAgent in June.

ChatGPT ran the other way. Its live-fetch agent made 6% fewer requests than in the first quarter, while ChatGPT referrals to websites grew 17% and were "upwards of 80%" of all AI referral traffic.

DataDome also says 54% of its customers had adopted an "agent trust policy", a written rule set for which automated clients may reach which parts of a site.

News UK, six months to September 2026. Its chief technology officer Tom Jackson told Press Gazette's conference that about a fifth of the publisher's traffic was non-human, and that about a third of that was blocked as "malicious bot traffic", as reported on 23rd Sep.

The three do not add up to one number. 51Degrees counts sessions on mixed-industry sites, DataDome counts requests on enterprise sites, and News UK is one publisher's own logs.

Wikimedia Foundation, 5th Oct 2026. Its investigation found agents it believes OpenAI operates made "millions of automated requests" to its public APIs and crawled millions of pages, mainly on Wikidata and Wikimedia Commons.

They also made "hundreds of thousands" of Wikidata Query Service queries, which "may have contributed to a partial outage" in May. The attribution is Wikimedia's belief, not an OpenAI confirmation, and the post gives no share of total traffic.

The same post reports sandbox test edits and failed attempts to use a hosted note-taking tool as a proxy. Wikimedia found no evidence its systems or data were compromised.

The policy response is under way. More than 300 news executives went to Washington on 29th Sep for the Stealth Bot Prohibition Act, which would require crawlers to identify themselves and disclose their purpose, as Search Engine Land reported.

Press Gazette's piece also carries a robots.txt figure from the analytics firm Known Agents: publishers block about 21% of AI scraper bots in their files. The table above is the list to check yours against.

OpenAI publishes ranges for each bot at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. Anthropic publishes claude.com/crawling/bots.json.

Perplexity publishes perplexity.com/perplexitybot.json and perplexity-user.json. DuckDuckGo publishes duckduckgo.com/duckassistbot.json, and Amazon publishes its list at developer.amazon.com/amazonbot/ip-addresses.

Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot" and recommends reverse DNS checks.

ByteDance publishes no IP list for Bytespider that we could find. A request carrying that string cannot be checked either way.

# robots.txt: the training crawlers documented above
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
Disallow: /

That block leaves every search crawler alone. Pages stay eligible for ChatGPT search, Perplexity, Claude search, Meta AI search and DuckAssist.

Bing needs the meta tags instead.

Cloudflare said on 16th Oct 2026 that it now measures "how often a crawler respects site preferences" for its Verified bots. The first signals are "Does the bot respect instructions?" and "Did it respond to direct feedback?"

The observations become "an explainable trust band" per bot. Cloudflare will share the scores "publicly on Radar".

It plans to let customers "choose the trust threshold". It names no scores and no date.