Firewall by purpose, not by name

The order of operations

1. Verify. Is this request really from who it claims? If you cannot tell, treat the name as noise.
2. Classify. Is it training, search, a user-triggered fetch, or an agent?
3. Decide. Choose a policy per class, based on what that class gives you.
4. Enforce at the right layer. robots.txt for bots that listen, the firewall for the rest.

Why a User-Agent rule is not enough

A User-Agent is a sentence the client writes about itself. Google, Common Crawl and Perplexity all warn that their names get spoofed. Cloudflare documented an undeclared crawler using a plain Chrome string on macOS at 3 to 6 million requests a day, from addresses outside the published ranges, and attributed it to Perplexity, which disputed the findings. [Cloudflare 2025]

A name rule still has a job: it cheaply stops bots that are honest about who they are. Just do not mistake it for a security control.

Three ways to prove identity

1. Published IP lists

OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple, Common Crawl, DuckDuckGo and Mistral publish address files in a shared JSON shape. We fetch them all hourly. Meta publishes none on its crawler page, and Amazon publishes its ranges as HTML.

2. Reverse DNS, then forward DNS

Google, Bing, Apple and Common Crawl support this. You need both directions, because anyone can set a reverse record on an address they control.

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

3. Web Bot Auth signatures

This is the only method that survives address rotation. The bot signs each request, and you check the signature against a public key directory. ChatGPT's operator signs its requests, and Google says it is experimenting with the identity https://agent.bot.goog. Cloudflare and Vercel both verify these at the edge, and the protocol is now an IETF working-group draft.

Try the verifier on an address from your own logs.

What each decision costs

These are the policies people get wrong most often.

• Blocking Googlebot takes you out of Google Search. To opt out of Gemini training, disallow Google-Extended instead. It has no effect on Search. AI Overviews are served from the Search crawl, so there is no crawler you can block to leave those alone.
• Blocking OAI-SearchBot removes you from ChatGPT search answers. Blocking GPTBot only stops training.
• Blocking PerplexityBot removes you from Perplexity citations.
• Blocking Applebot removes you from Spotlight and Siri. Applebot-Extended is the training switch.
• Blocking by network also hits real services hosted there, including monitoring, webhooks and your own integrations.

What robots.txt can and cannot do

The training and search crawlers of the major operators say they honour robots.txt. User-triggered fetchers often do not, by design. OpenAI says rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores” them. TollBit reported that 42% of ChatGPT-User fetches landed on sites blocking it. [TollBit 2026]

Changes also take time. Caches run 24 hours for OpenAI and Meta, 72 hours for a DuckDuckGo opt-out, and up to 30 days for Amazon.

Two newer lines express preferences rather than access rules: Content-Usage: train-ai=n from the IETF AIPREF drafts, and Cloudflare's Content-Signal. They cost nothing to add. Google says it will support an AIPREF signal in early 2027.

When the traffic does not identify itself

The worst scraping no longer looks like a datacenter. LWN describes millions of residential and mobile IPs making two or three requests each. [LWN 2026] A per-IP rate limit never fires against that. What helps:

• Limit on a composite key. Use network, TLS fingerprint (JA4) and path class together, rather than IP alone.
• Watch your expensive paths. Over 51% of AI requests reach origin, against under 9% of human ones. [Fastly 2026] Search pages, faceted filters and full repository views are where the bill comes from.
• Serve what caches. Cloudflare says over half of AI crawler traffic re-fetches unchanged pages. [TechCrunch 2026] Good ETag and Last-Modified handling turns those into cheap 304s.
• Try 402. Returning HTTP 402 on paths you do not want crawled is cheap. Stack Overflow reported that many unauthorized crawlers simply stopped, though that report is second-hand.

Proof of work and tarpits, honestly

Anubis stops naive crawlers, and it costs anonymous readers something too. LWN sees signs that it is being bypassed, and proof of work is weak against work spread over millions of machines. Use it on endpoints that are expensive to render, not as your front door.

Tarpits such as Nepenthes and AI Labyrinth are better at detecting than deterring, and Cloudflare concedes that bots now look for hidden links. Anything that follows a hidden link is a bot, which is useful information whether or not it wastes the bot's time.

Starting rules

(http.user_agent contains "Googlebot" and not cf.client.bot)

map $http_user_agent $training_bot {
    default 0;
    ~*(GPTBot|ClaudeBot|CCBot|Bytespider) 1;
}

server {
    if ($training_bot) { return 403; }
}

On Vercel, turn on the AI Bots managed ruleset in log mode first and read a week of results before denying anything. Bot Protection excludes verified bots, but Vercel's docs warn that it breaks behind a reverse proxy. On Cloudflare, AI Crawl Control gives per-crawler decisions on every plan.

Whatever you choose, log before you block. A week of evidence is worth more than a clever rule.