Botservatory

Monitor

Your own sky, verified

The same instrument that watches this site, pointed at yours. Every visitor that claims to be a crawler is checked against what its operator publishes, so you see who is real and who is borrowing a name. Free for one site, with no account.

Add it to a Next.js site

Two small changes. The file filters out ordinary browsers before anything leaves your server, so a busy site sends one small request per automated visit, in the background, without slowing the page.

lib/botservatory.ts
// lib/botservatory.ts: reports automated requests to Botservatory. Ordinary browsers are never sent.
import type { NextFetchEvent, NextRequest } from "next/server";

const BOT_HINT = /bot|crawl|spider|slurp|scrape|preview|fetcher|curl|wget|python|httpx|aiohttp|Go-http|okhttp|Java\/|node-fetch|undici|axios|Headless|Playwright|Puppeteer|PhantomJS|Scrapy|GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-User|Claude-SearchBot|Googlebot|Google-CloudVertexBot|GoogleOther|Google-Agent|Google-GeminiNotebook|bingbot|PerplexityBot|Perplexity-User|meta-externalagent|meta-externalfetcher|meta-webindexer|meta-externalads|facebookexternalhit|Amazonbot|Amzn-SearchBot|Amzn-User|Applebot|CCBot|DuckAssistBot|MistralAI-User|Bytespider|cohere-ai|Diffbot|xAI-Grok|HeadlessChrome|Playwright|Puppeteer|PhantomJS|\bcurl\/|\bWget\/|python-requests|python-httpx|aiohttp|Scrapy|Go-http-client|node-fetch|undici|axios|okhttp|Java\/|AhrefsBot|SemrushBot|MJ12bot|DotBot|PetalBot|YandexBot|Baiduspider|DuckDuckBot|UptimeRobot|Pingdom|Better ?Stack|bot\b|crawler|spider|scraper/i;

export function observeBots(request: NextRequest, event: NextFetchEvent): void {
  const key = process.env.BOTSERVATORY_KEY;
  if (!key) return;
  const ua = request.headers.get("user-agent") ?? "";
  const signatureAgent = request.headers.get("signature-agent");
  if (ua && !signatureAgent && !BOT_HINT.test(ua)) return;
  // x-real-ip is set by the platform; otherwise only the last X-Forwarded-For hop (added by
  // your nearest proxy) is trusted. Earlier entries come from the client and can be forged.
  const ip = request.headers.get("x-real-ip")?.trim() || request.headers.get("x-forwarded-for")?.split(",").map((hop) => hop.trim()).filter(Boolean).pop() || null;
  event.waitUntil(
    fetch("https://botservatory.ai/api/v1/ingest", {
      method: "POST",
      headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
      body: JSON.stringify({ events: [{ ua, ip, path: request.nextUrl.pathname, t: Date.now(), signatureAgent }] }),
      signal: AbortSignal.timeout(3000),
    }).then(() => undefined, () => undefined),
  );
}
proxy.ts
// proxy.ts on Next.js 16, middleware.ts before it. A default export works under either name.
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
import { observeBots } from "./lib/botservatory";

export default function handler(request: NextRequest, event: NextFetchEvent) {
  observeBots(request, event);
  return NextResponse.next();
}

Already have a proxy or middleware? Add the observeBots(request, event) line to it. Any other stack can send the same JSON to POST https://botservatory.ai/api/v1/ingest; see the API.

What you get, and what we keep

The free tier. One site, seven days of history, and 100,000 automated requests a month. After that, requests are refused until the next month begins and your site is unaffected.

What is stored. For each automated request: the time, the path, which bot it claimed to be, and whether that checked out. Visitor addresses are used to verify and then dropped; they are never written down, whole or hashed. Requests from ordinary browsers are not sent at all.

Who can see it. Only someone with your dashboard link. It is not listed or indexed. Lose the link and the data cannot be recovered; start again with a new site.