# Botservatory

> Botservatory watches which bots crawl the web, verifies who they really are, and explains what to block, what to allow, and how to benefit from the difference.

No major AI operator has committed to reading this file. We publish it anyway and count who asks for it at https://botservatory.ai/now.

## Guides
- [How to firewall bots correctly](https://botservatory.ai/firewall): verify, classify, decide, then enforce
- [How to make bot traffic work for you](https://botservatory.ai/benefit): which bots send visitors, and how to be cited
- [How to tell a bot from a person](https://botservatory.ai/bot-or-not): three classes of visitor and the signals that still work

## Reference
- [Crawler catalogue](https://botservatory.ai/catalogue): every major crawler and how to prove its identity
- [Numbers and sources](https://botservatory.ai/numbers): every statistic, with its population and date
- [Ephemeris](https://botservatory.ai/ephemeris): a dated timeline of events

## Tools
- [Bot verifier](https://botservatory.ai/tools/verify): test an IP and User-Agent against operator data
- [robots.txt generator](https://botservatory.ai/tools/robots): separate training, search and fetchers

## Data
- [Catalogue JSON](https://botservatory.ai/api/v1/catalogue.json): CC BY 4.0
- [Live observations JSON](https://botservatory.ai/api/v1/now.json): this site's traffic only
- [OpenAPI description](https://botservatory.ai/api/openapi.json)