Observing since November 2026
Most of your visitors aren't reading. They're collecting.
Botservatory watches who is really crawling the web, checks their identity, and shows you what to let in, what to keep out, and what each decision costs.
- Training
- Search index
- User fetch
- Agent
- Automation
53%+
Share of web traffic that is bots, calendar 2025
Thales 2026 Bad Bot Report, May 5, 2026
6.5×
How much faster AI requests grew than human traffic, January to May 2026. AI requests rose about 30%
Fastly, Jun 9, 2026
51% vs 9%
Share of requests that reach origin: over 51% of AI requests, under 9% of human requests
Fastly, Jun 9, 2026
−40.2%
Google Search referrals to publishers, July 2025 to July 2026, after −21.9% the year before
Chartbeat, via Search Engine Journal, Sep 24, 2026
Firewall it correctly
Verify first, then decide by purpose. A User-Agent string is a claim, and blocking the wrong name can cost you Google.
Make bots work for you
Some bots send people. Learn which ones, how to be cited, and what is known about getting paid for the rest.
Tell what is a bot
There are three classes now, not two. Some of the most capable visitors are agents with a real person behind them.
The logbook
Full sky →| When | Visitor | Class | Identity | Path |
|---|---|---|---|---|
| ClaudeBot | Training | Verified by IP list | / | |
| ClaudeBot | Training | Verified by IP list | / | |
| ClaudeBot | Training | Verified by IP list | /robots.txt | |
| Empty User-Agent | Automation | Self-described | /.env | |
| Empty User-Agent | Automation | Self-described | /.env | |
| OAI-SearchBot | Search index | Verified by IP list | /robots.txt | |
| OAI-SearchBot | Search index | Verified by IP list | /robots.txt | |
| Unlisted bot | Automation | Self-described | / |
Observed on botservatory.ai only. This is not a measure of the wider web.