Claude Updates: Web Search, Crawlers and Anthropic News

Claude as a search surface: what its crawlers do, what traffic it sends, and every Anthropic change that matters to site owners.

Last updated: Last checked: Web search global on all plans; Claude for Chrome on all paid plans; the $1.5B authors settlement approved
+386%Claude referral growth, Jan to Apr 2026SE Ranking panel of 101,574 sites
86.7%Citation overlap with Brave SearchProfound, a March 2025 measurement
$1.5BAuthors settlement, approved Jul 2026About $3,000 per work across ~500,000 works
3Documented Anthropic crawlersAll three honor robots.txt, per Anthropic

This page tracks Claude, Anthropic's assistant, as a search surface: its web search and citations, the Claude for Chrome agent, the three documented crawlers and what independent measurement says about the traffic they cost and send.

Anthropic announces less than any company on this beat, so much of what is known comes from measurement: referral panels, crawl-ratio studies and bot telemetry. Where a figure appears here, we say what it counts and when it was measured, because Claude numbers age fast and often measure opposite directions of traffic.

We update within a day or two of significant changes, and every claim links to a source we have read.

Claude search timeline: March 2025 to today

Every significant Claude search-surface change since web search launched, newest first. A thinner timeline than its neighbors is honest for this beat: Anthropic ships and announces less, and measurement fills the gaps.

One dot per milestone since Claude web search launched in March 2025, one lane per event type. Hover a dot to identify it; most dots click through to the entry below.
20252026Product2Availability3Data3Legal1
DateEventTypeWhy it matters
2026
$1.5B authors settlement approvedLegalBelieved the largest US copyright settlement; no appellate precedent set
SE Ranking measures Claude referrals: +386% off a tiny baseData1.40% of AI-referred traffic in a 101,574-site panel
2025
Claude for Chrome reaches all paid plansAvailabilityPro, Team and Enterprise join after months of testing
Claude for Chrome opens to all Max subscribersAvailabilityScheduled tasks and multi-tab workflows added since the pilot
Cloudflare publishes the crawl-to-refer seriesDataAnthropic measured at the highest crawl cost per referral of any AI operator
Claude for Chrome pilot beginsProduct1,000 Max users; published adversarial safety numbers
Web search goes global on all plansAvailabilityFree tier included, via an update note rather than an announcement
Search backend identified as BraveDataSubprocessor listing plus two independent measurements
Claude web search launchesProductPaid US preview on Claude 3.7 Sonnet, with direct citations

Latest updates

Legal

Anthropic's $1.5 billion authors settlement gets final approval

Judge Araceli Martinez-Olguin approved the $1.5 billion class settlement on 20th July, per TechCrunch: "The payout will deliver $3,000 per work across an estimated 500,000 works, shared among the authors and publishers who hold rights to them."

The legal shape matters as much as the number. Judge William Alsup had ruled that TRAINING on the books was fair use, while downloading and storing millions of pirated copies was not, and granted preliminary approval on that basis in 2025.

Because Anthropic settled rather than appealed, no binding appellate precedent was set in either direction. Every other AI copyright docket, including Perplexity's eleven cases, proceeds without a controlling answer.

For site owners the practical reading is economic: pirated training corpora now carry a price tag believed to be the largest in US copyright history, which strengthens the hand of anyone licensing content and the case for controlling the training crawler.

Study

SE Ranking: Claude referrals grew 386% in four months, from a tiny base

SE Ranking's study measured Claude referrals across 101,574 websites with Google Analytics in 250 countries, over a window running January 2025 to April 2026.

The growth is the headline: Claude's share of the panel's referral traffic went from 0.0029% in January 2026 to 0.0141% in April, a 386% rise in four months.

The base is the caveat. All AI platforms combined drove 0.33% of the panel's traffic, and within that Claude held 1.40%, against ChatGPT's 78.23%, Perplexity's 9.33%, Gemini's 6.85% and Copilot's 3.57%, per Search Engine Journal's write-up of the study.

Two disclosures belong with the numbers. The study counts only direct clicks from AI platforms, not AI-influenced visits arriving through other channels. And SE Ranking sells AI visibility tooling, a point SEJ's coverage makes alongside the observation that Claude is used mainly for writing, coding and analysis rather than search.

Product

Claude for Chrome reaches every paid plan

Anthropic's Claude for Chrome page carries the arc in its own update notes: a pilot limited to 1,000 Max plan users from 25th August 2025, all Max subscribers from 24th November, and then "Update: Now available to Pro, Team, and Enterprise plans (Dec 18, 2025)", alongside "After months of real-world testing, we're ready to expand to all paid plans".

The extension lets Claude act in the browser: navigating, filling and completing tasks across tabs, with scheduled tasks and multi-tab workflows added during the beta.

The safety disclosure is unusually concrete. Anthropic published adversarial results: "Browser use without our safety mitigations showed a 23.6% attack success rate when deliberately targeted by malicious actors." With autonomous-mode mitigations that fell to 11.2%, and a browser-specific set of four attack types went from 35.7% to zero.

Claude for Chrome outlived the standalone AI browsers: OpenAI's Atlas shut down in August 2026 and Microsoft retired Edge's Copilot Mode in May, both refocusing on exactly this extension-plus-app shape.

Study

Cloudflare's crawl-to-refer series: Anthropic crawls tens of thousands of pages per referral

Cloudflare's methodology, from its first report: HTML requests from a platform's crawlers, divided by HTML requests arriving with that platform's hostname as the referrer, normalized to one referral.

Anthropic's ratios were the series' extreme: roughly 71,000:1 measured in late June 2025, 38,065.7:1 in July (down 86.7% from January's 286,930:1), and roughly 50,000:1 in the first week of August. The July comparison set: OpenAI 1,091.4:1, Perplexity 194.8:1, Google 5.4:1, Microsoft 40.7:1.

For publishers, the August report's sector cut is the relevant row: in news and publications, Anthropic crawled about 2,500 pages per referral, OpenAI 152 and Perplexity 32.7.

Cloudflare also split crawling by purpose in the July data: about 80% training, 17% search, 3.2% user actions.

One stated limitation keeps the Anthropic number honest: traffic referred by Claude's native app carries no referrer header, so its ratio is overstated by an unknown amount. And a dating note: newer 2026 ratios circulate on SEO sites without any Cloudflare publication behind them, so this page carries the 2025 series until Cloudflare publishes again.

Archive: 3 older entries
Availability

Claude web search goes global on every plan

The gate fell quietly, in an update note on the launch post: "Update: Web search is now available globally on all Claude plans. (May 27, 2025)".

That completed a two-month rollout that had started US-only for paid users in March, and it made citations-bearing web answers a default Claude capability rather than a feature-preview extra.

For site owners the date marks when Claude traffic became possible at global scale, though the measured volumes stayed tiny well into 2026.

Data

Claude's search backend identified as Brave, by three independent signals

Anthropic never announced what powers Claude's web search. The identification came from three directions within days of launch.

Anthropic added Brave Search to its subprocessor list on 19th March 2025. Simon Willison then observed that Claude's ten citations for a test query "were an exact match for that search on Brave", and that Claude's search function exposes a parameter named BraveSearchParams.

Profound quantified it: "Claude leverages Brave Search as its primary search backend, confirmed by a statistically significant overlap of 86.7% (13 out of 15 total results) between Claude's cited results and Brave Browser's top non-sponsored results." Their contrast figure: ChatGPT-to-Bing alignment of only 26.7%.

Handle the number with its method: 13 of 15 results across three test queries, on one day in March 2025. It is directional evidence about the then-current integration, never confirmed by Anthropic beyond the subprocessor listing, and we know of no newer measurement.

If it still holds, the SEO consequence is unusual and testable: visibility in Brave's organic results would be the upstream lever for Claude citations.

Product

Claude gets web search, with direct citations

The launch post set the scope plainly: "Web search is available now in feature preview for all paid Claude users in the United States. Support for users on our free plan and more countries is coming soon."

The citation promise is the part site owners care about: "When Claude incorporates information from the web into its responses, it provides direct citations so you can easily fact check sources."

The launch model was Claude 3.7 Sonnet. Within a week the backend had been identified as Brave Search, and by late May the feature was global on every plan.

Anthropic's three crawlers, and what blocking each one does

Anthropic is the only AI search operator claiming robots.txt compliance for its user-triggered fetcher.

Anthropic documents three bots, and its compliance claim is the strongest of any operator on this beat: "Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt." That claim covers all three, including the user-triggered fetcher, where OpenAI and Perplexity both document that robots.txt may not or does not apply to theirs.

CrawlerWhat it does, per AnthropicWhat blocking it means
ClaudeBotCollects web content "that could potentially contribute to their training" of generative modelsExcludes future materials from training datasets; the training opt-out
Claude-UserFetches a site when an individual asks Claude about itBlocks retrieval for user queries; Anthropic warns it "may reduce your site's visibility for user-directed web search"
Claude-SearchBot"Navigates the web to improve search result quality for users"Prevents indexing that feeds Claude's search relevance

Operational details from the same documentation: Anthropic says its bots respect anti-circumvention technologies rather than bypassing CAPTCHAs, supports the non-standard Crawl-delay robots extension, and warns that IP-based blocking "may not work correctly" because the bots run from cloud IPs. Anthropic does not publish full user-agent strings on that page; the tokens above are the robots.txt names.

These are the operator's claims about itself. The independent counterweight lives in the measurement section: TollBit's network found 30% of AI bot scrapes in Q4 2025 bypassed explicit robots.txt permissions across the industry, which is why serious enforcement pairs robots.txt with firewall rules.

Anthropic's documentation carries no revision date, so this table is dated to our fetch of it on 13th August 2026.

The measurement layer: what AI surfaces cost and send

The cross-surface studies that quantify AI crawling and referrals, each labeled with what it counts and when.

Nobody announces this data; it is measured by infrastructure and analytics companies, and it is the closest thing the AI-search beat has to a shared reality. Every row names what it counts, because these figures are routinely misquoted with the wrong object.

FindingWhat it countsSource and date
+386% Claude referral growth Jan to Apr 2026 (0.0029% to 0.0141% of panel traffic); all AI platforms combined 0.33%Referral visits arriving FROM AI platforms, 101,574-site Google Analytics panelSE Ranking, 3rd Jun 2026
Claude ~9% of generative-AI web traffic (from ~2%); ChatGPT ~53% (from ~76%); category 9.5B monthly visits, +70% YoYVisits TO the AI platforms themselves, worldwide panelSimilarweb, updated 29th Jul 2026
Anthropic crawl-to-refer 38,065.7:1 (Jul 2025); OpenAI 1,091.4:1; Perplexity 194.8:1; Google 5.4:1; news-sector cut 2,500:1 / 152:1 / 32.7:1Crawler HTML requests per one referred HTML visit, Cloudflare network; Claude app referrals carry no referrer, overstating AnthropicCloudflare, Jul and Aug 2025 reports
AI crawling by purpose: ~80% training, ~17% search, 3.2% user actionsShare of AI bot requests by operator-declared purposeCloudflare, 29th Aug 2025
1 AI bot visit per 31 human visits (Q4 2025, from 1 in 50 in Q2); robots.txt bypass 3.3% (Q4 2024) to ~13% (Q1 2025) to 30% (Q4 2025), worst agent ChatGPT-User at 42%; citation CTR 0.8% to 0.27%Bot visits, bypass share and citation click-through across TollBit's publisher networkTollBit State of the Bots via Digiday, 11th Feb 2026
17.7B AI agent requests in Q2 2026, +45% QoQ; Meta's agents now the majority of AI agent traffic; ChatGPT commands roughly 80 to 88% of AI referralsAI agent HTTP requests across DataDome's customer network; referral share is a separate measure in the same reportDataDome via the wire release, 16th Jul 2026
Only 12% of links cited by ChatGPT, Gemini and Copilot rank in Google's top 10 for the same prompt; Perplexity highest at 28.6%Cited URLs checked against Google's top 10, 15,000 long-tail queries, Ahrefs indexAhrefs, 11th Aug 2025
ChatGPT cites ~15 sources per response vs Gemini's ~3; 126M+ US prompts analyzedAverage cited sources per response, four platforms, Jan to Apr 2026 windowSemrush AI Visibility Index press release, 26th Jun 2026

A note on vintages: newer versions of several of these series exist behind walls we could not read (TollBit's 2026 edition is demo-gated, Cloudflare's live Radar page renders client-side only), and the 2026 crawl-ratio numbers circulating on SEO blogs have no verifiable publication behind them. This table advances only when a primary does.

What Claude means for your site today

Tiny referrals, real crawl cost, honest controls, and one testable lever nobody else has.

The honest summary: Claude sends very little traffic (1.40% of an already-small AI referral pie), costs real crawl budget (the worst measured crawl-to-refer ratio of any operator), and offers cleaner controls than its rivals (three robots-honoring crawlers with separable purposes).

The control decisions map cleanly. Allow Claude-SearchBot and Claude-User if you want visibility in Claude's answers; their opt-outs cost you retrieval and search presence. Decide ClaudeBot separately, because it is the training crawler, and the settlement just priced what training data is worth.

The distinctive lever is the Brave backend: if the March 2025 identification still holds, Claude citations follow Brave's organic rankings, which makes Brave Search Webmaster-style hygiene a testable input in a way no other AI surface offers.

Beyond that, our guidance matches the rest of this section: citable, specific pages earn AI mentions everywhere, referral volume expectations should stay low, and any tool promising "Claude optimization" is selling ahead of the evidence.

Frequently asked questions

Does Claude search the web?

Yes. Web search launched in March 2025 as a US paid preview and has been available globally on every Claude plan, free included, since 27th May 2025. When Claude uses the web it shows direct citations, which Anthropic promised at launch.

What search engine does Claude use?

Brave Search, per three independent signals from March 2025: Anthropic listed Brave as a subprocessor, Simon Willison found Claude's citations exactly matching Brave results, and Profound measured 86.7% overlap with Brave's top organic results. Anthropic has never confirmed it publicly, and the measurements date to launch, so we treat it as the identified backend as of those probes.

How do I block, or allow, Claude's crawlers?

Three robots.txt tokens with separable jobs: ClaudeBot (model training), Claude-SearchBot (search indexing) and Claude-User (fetching when a user asks about your site). Anthropic says all three honor robots.txt, uniquely including the user-triggered one, supports Crawl-delay, and warns IP blocking may misfire because the bots use cloud IPs.

How much traffic does Claude actually send?

Very little, growing very fast. In SE Ranking's 101,574-site panel, Claude referrals grew 386% between January and April 2026 yet still reached only 0.0141% of total traffic, 1.40% of AI-referred visits. ChatGPT accounts for around four-fifths of AI referrals in every panel we track.

What was the Anthropic authors settlement?

A $1.5 billion class settlement over pirated books used in training, approved 20th July 2026, paying about $3,000 per work across an estimated 500,000 works. The court had ruled training itself fair use while the pirated copies were not, and because Anthropic settled, no appellate precedent was set.

What is Claude for Chrome?

Anthropic's browser agent: a Chrome extension where Claude navigates and completes tasks across tabs. It ran as a 1,000-user pilot from August 2025 and has been available on every paid plan since 18th December 2025. Anthropic published its adversarial safety numbers, 23.6% attack success without mitigations falling to 11.2% with them.

Is Claude big enough to matter for SEO?

As a referrer, not yet: its measured share is tiny. As a crawler, yes today: Cloudflare measured Anthropic at the highest crawl-to-refer ratio of any AI operator, thousands of pages crawled per referral in the news sector. The control decision therefore matters now, while the traffic case is a bet on the growth curve.

Spotted a change we have not covered? Ranking volatility, a new SERP test, a platform change: email us a tip and we will credit you if we cover it.
Cite this page: Keywords Everywhere, "Claude Updates: Web Search, Crawlers and Anthropic News", last updated 13th Aug 2026, https://keywordseverywhere.com/news/claude-updates/