Appendix
Numbers and sources
−40.2%
Google Search referrals to publishers, July 2025 to July 2026, after −21.9% the year before
Measured on: Chartbeat publisher network.
Chartbeat, via Search Engine Journal, Sep 24, 2026 · Confirmed
0.01%
AI referrals as a share of publisher traffic, up eightfold in two years
Measured on: Chartbeat publisher network.
Chartbeat, via Search Engine Journal, Sep 24, 2026 · Confirmed
Nearly half
Share of internet traffic generated by bots
Measured on: Vercel's documentation, no method given.
Vercel bot management docs, Sep 10, 2026 · Confirmed
2–3
Requests per IP address in the most harmful scraping, spread across millions of residential and mobile proxies
Measured on: LWN's account of its own traffic and reports from others.
LWN, Jul 10, 2026 · Confirmed
50%+
Share of AI crawler traffic that re-fetches unchanged pages, per Cloudflare
Measured on: Cloudflare statement.
TechCrunch, Jul 1, 2026 · Confirmed
6.5×
How much faster AI requests grew than human traffic, January to May 2026. AI requests rose about 30%
Measured on: Fastly customer cohort.
Fastly, Jun 9, 2026 · Confirmed
85 / 15
Crawlers versus fetchers in AI bot traffic, May 2026
Measured on: Fastly customer cohort.
Fastly, Jun 9, 2026 · Confirmed
51% vs 9%
Share of requests that reach origin: over 51% of AI requests, under 9% of human requests
Measured on: Fastly customer cohort, May 2026.
Fastly, Jun 9, 2026 · Confirmed
+555%
Growth in Claude-related traffic over the January 2026 baseline
Measured on: Fastly customer cohort.
Fastly, Jun 9, 2026 · Confirmed
Flat
AI crawler activity shows no day and night curve across 24 hours
Measured on: Fastly customer cohort.
Fastly, Jun 9, 2026 · Confirmed
40%
Share of bot traffic classed as malicious
Measured on: Thales/Imperva customer base.
Thales 2026 Bad Bot Report, May 5, 2026 · Confirmed
12.5×
Growth in AI-driven bot attacks
Measured on: Thales/Imperva customer base. Method is in a gated PDF we have not read.
Thales 2026 Bad Bot Report, May 5, 2026 · Confirmed
27%
Share of attacks aimed at APIs and identity systems
Measured on: Thales/Imperva customer base.
Thales 2026 Bad Bot Report, May 5, 2026 · Confirmed
Under 15
Sites among the top 200,000 domains exposing MCP server cards or API catalogs. About 4% declare AI preferences in robots.txt
Measured on: Cloudflare Agent Readiness scan.
WebHosting.Today, Apr 22, 2026 · Second-hand
1 in 31
AI bots as a share of site visits by Q4 2025, up from 1 in 200 nine months earlier
Measured on: TollBit's network. The report is gated, so this comes from press coverage.
TollBit, via press, Feb 2026 · Second-hand
42%
Share of ChatGPT-User fetches that landed on sites blocking it in robots.txt
Measured on: TollBit's network. The report is gated, so this comes from press coverage.
TollBit, via press, Feb 2026 · Second-hand
1.2–9×
How much better AI referrals convert than organic search. The range spans several studies, none of them primary research
Measured on: Marketing studies with mixed methods.
Visionary Marketing roundup, 2026 · Second-hand
10.13%
Share of 300,000 domains with an llms.txt file
Measured on: SE Ranking sample.
SE Ranking, via DEV, 2026 · Second-hand
80 / 18 / 2
Crawling by purpose: training, search and user actions
Measured on: Cloudflare network, trailing 12 months to mid-2025.
Cloudflare, Aug 29, 2025 · Confirmed
38,066 : 1
Anthropic crawls per referred visit, July 2025. OpenAI was 1,091:1, Perplexity 195:1 and Google 5.4:1. Later 2026 figures circulate only second-hand and disagree
Measured on: Cloudflare network, July 2025.
Cloudflare, Aug 29, 2025 · Confirmed
3–6M
Daily requests from an undeclared crawler Cloudflare attributed to Perplexity. Perplexity disputed the findings
Measured on: Cloudflare network.
Cloudflare, Aug 4, 2025 · Confirmed