Botservatory

← Catalogue

Common Crawl

CCBot

An open archive, used heavily for AI training. Common Crawl warns about impersonators.
Class
Training
robots.txt
Says it honours robots.txt
Evidence
Confirmed from operator documentation
Source
Published User-Agent
CCBot/2.0 (https://commoncrawl.org/faq/)

Proving it is real

  • Check the IP against Common Crawl's CCBot file.
  • Run a reverse DNS lookup, confirm the name ends in .crawl.commoncrawl.org, then confirm it resolves back to the same IP.

Check an address now.

Blocking it

robots.txt
User-agent: CCBot
Disallow: /
Cloudflare custom rule expression
(http.user_agent contains "CCBot")
nginx
if ($http_user_agent ~* "CCBot") {
    return 403;
}

Matching a name only blocks bots that tell the truth about it. Before you do this, read what each decision costs.

Last reviewed Nov 2, 2026