Field guide 03

Bot or not is the wrong question

Vendors put bots anywhere from “nearly half” to over 53% of traffic. Both can be right, because they measure different things. The category itself is also breaking down.

How much of the web is bots?

Thales says over 53%, measured across its customers. [Thales 2026] Vercel's documentation says nearly half. [Vercel 2026] Neither is “the web”. Each is one company's view of its own traffic, so when you see one number quoted alone, ask who was measured.

Three classes, not two

Declared automation

It tells you what it is. GPTBot and Googlebot are examples. You can verify many of them, and the catalogue covers the major ones.

Undeclared automation

It claims to be a browser. This is where the harm tends to sit. Thales classes 40% of bot traffic as malicious, [Thales 2026] and 27% of attacks target APIs and identity systems. [Thales 2026]

An agent acting for a person

A real human asked for this, and software is doing it, in a full browser, with their session. Comet is Chromium and reportedly looks like ordinary Chrome. ChatGPT's agent reportedly sends a plain Chrome User-Agent and identifies itself only by signature. Treating this as an attack can turn away a customer who is trying to buy from you.

Good and bad are about purpose

No bot is good in the abstract. Googlebot is welcome until you object to AI Overviews. GPTBot is harmless to your referrals and useless to them. ChatGPT-User ignores your preferences, and it is also the one with a person waiting on the other end. Judge a bot by what it is doing and what it gives back.

Signals that still work

Signal	Why it works	Where it fails
Identity checks out	Declared name matches an IP list, reverse DNS or a signature	Only covers operators who publish
No day and night rhythm	AI crawlers are flat across 24 hours, while people sleep	Needs a day of data per cluster
Asks for robots.txt first	People never request it	Stealth crawlers skip it
Follows a hidden link	No person can click it	Modern bots look for these
Skips your cache	Over 51% of AI requests reach origin, under 9% of human ones	Needs a CDN-level view
Comes from a datacenter	Cheap first pass	Residential proxies defeat it
Fingerprint contradicts claim	A curl TLS handshake claiming to be Chrome	Real headless Chrome matches
No assets, no JavaScript	The classic sign	Agentic browsers run everything

No single signal is enough. Combine them. The sources for the numbers above are Fastly on rhythm and Fastly on cache behaviour. Our dial plots each visit by hour, so you can look for the missing night yourself.

What your analytics miss

Client-side analytics only run where JavaScript runs. Training crawlers, most fetchers and plain HTTP clients never appear. Agents do appear, and they get counted as people. So your dashboard undercounts bots and overcounts humans at the same time. Server or edge logs are the only complete record.

Five minutes with your own logs

Who says they are a bot (nginx combined log)

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -30

Which addresses claim to be Googlebot

grep Googlebot access.log | awk '{print $1}' | sort -u > claimed.txt

Take any address from that list to the verifier. If it fails, something is using Google's name to get past your rules. That is often the most useful thing you will learn all week.

Last reviewed Nov 2, 2026