Botservatory

← Catalogue

Meta

meta-webindexer

Index for Meta AI.
Class
Search index
robots.txt
Says it honours robots.txt
Evidence
Confirmed from operator documentation
Source
Published User-Agent
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)

Proving it is real

Meta publishes no way to confirm this identity, so any request using this name has to be taken on trust.

Blocking it

robots.txt
User-agent: meta-webindexer
Disallow: /
Cloudflare custom rule expression
(http.user_agent contains "meta-webindexer")
nginx
if ($http_user_agent ~* "meta-webindexer") {
    return 403;
}

Matching a name only blocks bots that tell the truth about it. Before you do this, read what each decision costs.

Last reviewed Nov 2, 2026