{
  "id": 17259904,
  "title": "Blocking AI crawlers: robots.txt vs Cloudflare vs Anubis, and what each one actually does",
  "url": "https://urgent.news/2026/10/30/blocking-ai-crawlers-robots-txt-vs-cloudflare-vs-anubis-and-what-each",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-30T17:33:08.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/rozoomnews/blocking-ai-crawlers-robotstxt-vs-cloudflare-vs-anubis-and-what-each-one-actually-does-4di3"
  },
  "original_language": "en",
  "account": "The article explores three methods for managing AI crawler behavior: robots.txt, Cloudflare, and Anubis. Robots.txt is a simple protocol that informs crawlers of disallowed paths, while Cloudflare offers enforcement capabilities when DNS is already utilized. Anubis, on the other hand, requires clients to perform a proof-of-work challenge to access the content, which can prevent scraping but may also allow some non-browser clients to pass through. The article highlights that robots.txt is an effective, low-cost option that can demonstrate a reservation of rights under the EU AI Act, but it may not always be obeyed by AI crawlers. Cloudflare provides a more robust enforcement mechanism but relies on third-party decisions about AI crawlers. Anubis offers a unique approach that challenges clients directly, but it may not effectively block all non-browser clients. Ultimately, the choice between these methods depends on the specific needs and priorities of the site owner.",
  "summary": "A reader asked this under our analytics article, and it is the right question to ask in that order: robots.txt , Cloudflare, Anubis. The three get discussed as alternatives. They are not. One publishes a preference, one enforces a decision at the edge, one makes the client pay for the page. Choosing between them means deciding which of those three things you actually want. We have fifteen days of…",
  "key_points": [
    "Robots.txt informs crawlers of disallowed paths",
    "Cloudflare enforces restrictions via DNS",
    "Anubis challenges clients with proof-of-work"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}