Botservatory

Ephemeris

What moved, and when

A dated record of the events that changed how bots and the web deal with each other, newest first.

  1. Web Bot Auth moves forward at the IETF

    draft-ietf-webbotauth-httpsig-protocol-01 is a working-group document, written by Cloudflare and Google authors, on the standards track. Cryptographic identity for bots is becoming the normal case.

    Source

  2. Chartbeat: Google referrals to publishers fall 40.2%

    The year-on-year drop follows a 21.9% fall the year before. AI referrals are 0.01% of publisher traffic, up eightfold in two years.

    Source

  3. Cloudflare's new default for mixed-use crawlers

    Announced in July: on ad-bearing pages, crawlers classed as Training or Agent are blocked by default for new domains and Free-plan zones, while Search stays allowed. We confirmed the announcement. That it shipped on this date comes from secondary coverage only.

    Source · Second-hand

  4. AIPREF gives robots.txt a Content-Usage rule

    draft-ietf-aipref-attach-05 defines Content-Usage: train-ai=n as a robots.txt rule and an HTTP header, updating RFC 9309. Google says it will honour an AIPREF signal in early 2027.

    Source

  5. ChatGPT Atlas stops working

    OpenAI's browser, announced for shutdown on 9 July, is folded into the ChatGPT desktop app and Codex. Several outlets agree, but OpenAI's help page could not be opened.

    Source · Second-hand

  6. LWN: scraping has moved to residential proxies

    Most harmful traffic now arrives through millions of residential and mobile IPs at two or three requests each. Google disrupted IPIDEA and, with the FBI, NetNut. Anubis is widespread and shows signs of being bypassed.

    Source

  7. Cloudflare announces Pay Per Use

    The successor to Pay Per Crawl pays publishers when content creates value in an AI answer, not only on fetch. Launch partners are Ceramic.ai and You.com.

    Source

  8. Fastly: AI traffic outgrows human traffic 6.5 times over

    AI requests rose about 30% from January to May. Over 51% of them reach origin, against under 9% of human requests.

    Source

  9. Thales: bots pass 53% of web traffic

    The 2026 Bad Bot Report also classes 40% of bot traffic as malicious and says AI-driven bot attacks grew 12.5 times.

    Source

  10. Cloudflare scans the web for agent readiness

    Of the top 200,000 domains, fewer than 15 expose MCP server cards or API catalogs, and about 4% declare AI preferences in robots.txt.

    Source · Second-hand

  11. Cloudflare adds Redirects for AI Training

    Verified training crawlers can be sent a 301 to the canonical page, alongside a new Agent Readiness score.

    Source

  12. Google-Agent joins Google's crawler list

    Agents hosted on Google infrastructure get their own token, IP file and an experimental Web Bot Auth identity.

    Source

  13. Stack Overflow serves HTTP 402 to crawlers

    Working with Cloudflare's pay-per-crawl, it returned 402 to specific crawlers and reported that many unauthorized ones simply stopped.

    Source · Second-hand

  14. Amazon sues Perplexity over Comet

    Amazon alleges that Comet agents misidentify themselves in the User-Agent header. This is an allegation, not a finding.

    Source · Second-hand

  15. RSL 1.0 is published

    Really Simple Licensing puts machine-readable licence terms behind a pointer in robots.txt, including pay-per-crawl and pay-per-inference.

    Source

  16. Cloudflare launches Content Signals

    A Content-Signal line in robots.txt states search, ai-train and ai-input preferences, rolled out to over 3.8 million managed domains.

    Source · Second-hand

  17. Cloudflare publishes the crawl-to-click gap

    Anthropic made 38,066 crawls per referral in July 2025, OpenAI 1,091 and Google 5.4. Training accounted for about 80% of crawling.

    Source

  18. Vercel verifies Web Bot Auth signatures

    Vercel's bot verification adds Web Bot Auth and confirms that ChatGPT's operator signs its requests.

    Source

  19. Cloudflare accuses Perplexity of stealth crawling

    Cloudflare reports an undeclared Chrome User-Agent, rotating IPs outside the published ranges, and 3 to 6 million requests a day. It delisted Perplexity as a verified bot. Perplexity disputed the findings.

    Source

  20. Cloudflare's Content Independence Day

    AI crawlers are blocked by default for new zones, and Pay Per Crawl enters private beta using HTTP 402.

    Source

  21. Open source maintainers fight back

    Anubis and Nepenthes spread as maintainers report spending much of their time on crawler mitigation.

    Source · Second-hand