Some bots bring people with them

The trade

Chartbeat measured Google Search referrals to publishers down 40.2% in a year. [Chartbeat 2026] AI referrals were 0.01% of publisher traffic, up eightfold in two years. [Chartbeat 2026] So AI referrals do not replace what was lost. Be sceptical of anyone who says they do.

What is true is that the crawl-to-visit deal differs enormously by class. In July 2025 Cloudflare measured Google at 5.4 crawls per referral, OpenAI at 1,091 and Anthropic at 38,066. [Cloudflare 2025] Those ratios have moved since, and only second-hand 2026 figures exist, but the shape holds: search sends people and training does not. About 80% of crawling was for training. [Cloudflare 2025]

A sensible default

• Allow search crawlers. OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, bingbot, Applebot. These can cite you.
• Allow user-triggered fetchers. ChatGPT-User, Claude-User, Perplexity-User. A person asked about your page, and that person may click.
• Decide on training as a business question. If you sell words, you may refuse. If you sell a product, being in the training data might help people know you exist. No study settles that yet, so it is your call.

Every major operator now lets you make those choices separately. Generate the file.

Make your pages readable by machines

1. Send the content in the first response. Most crawlers do not run your JavaScript. If your page is an empty shell until hydration, they see an empty shell.
2. Publish structured data. Schema.org JSON-LD for your organization, articles, products and FAQs is the one machine-readable layer with a long public record of use.
3. Keep important facts in text. Pricing inside an image or behind a tab that fetches on click is invisible to most fetchers.
4. Answer actual questions. Fetchers arrive with a question. A page that states the answer plainly is easy to cite.
5. Be easy to fetch. Check that your firewall is not challenging the bots you allowed. This is the most common accidental loss.

The opening for agents

Cloudflare's scan of the top 200,000 domains found fewer than 15 exposing MCP server cards or API catalogs, and about 4% declaring AI preferences in robots.txt. [WebHosting.Today 2026] Machine-readable pricing and documentation are still rare, so doing it now is cheap differentiation.

llms.txt: publish it, expect little

About 10% of domains in one sample have an llms.txt file. [SE 2026] No major operator has publicly committed to reading one, and Google's John Mueller has said no AI system currently uses it. It takes a few minutes to write, so this site publishes one and counts who actually asks for it. Our logs will be better evidence than anyone's opinion, including ours.

Measure it

Client-side analytics will not show you crawlers, and they will count agent sessions as people. Watch server logs and track referrers from chatgpt.com, perplexity.ai, claude.ai, gemini.google.com and copilot.microsoft.com. ChatGPT is also widely observed adding utm_source=chatgpt.com.

Several studies say AI referrals convert better than organic search, by anywhere from 1.2 to 9 times. [Visionary 2026] Treat that as a reason to measure your own, because the methods vary and the numbers come from marketing research.

Getting paid

The tools exist. The buyers mostly do not, yet.

• HTTP 402 and Cloudflare Pay Per Crawl. Now evolving into Pay Per Use, which pays when content creates value in an answer. Launch partners are Ceramic.ai and You.com.
• RSL. Licence terms in a machine-readable file, linked from robots.txt.
• TollBit. A marketplace between publishers and AI companies.

As of the latest reporting we could find, in April 2026, no large AI company had publicly committed to paying through 402. Set these up if they cost you little, but do not put the income in your forecast.