Anthropic

Claude-SearchBot

Class: Search index
robots.txt: Says it honours robots.txt; Anthropic says disabling it stops indexing for search optimization.
Evidence: Confirmed from operator documentation; Source

Proving it is real

robots.txt

User-agent: Claude-SearchBot
Disallow: /

Cloudflare custom rule expression

(http.user_agent contains "Claude-SearchBot")

nginx

if ($http_user_agent ~* "Claude-SearchBot") {
    return 403;
}

Matching a name only blocks bots that tell the truth about it. Before you do this, read what each decision costs.

Last reviewed Nov 2, 2026