{
  "id": 11310679,
  "title": "Who's actually blocking AI crawlers in Europe? A 185-site census (first dated point)",
  "url": "https://urgent.news/2026/10/01/whos-actually-blocking-ai-crawlers-in-europe-a-185-site-census-first",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-01T23:55:52.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/pennyforgehq/whos-actually-blocking-ai-crawlers-in-europe-a-185-site-census-first-dated-point-3a59"
  },
  "original_language": "en",
  "account": "A census of 185 European websites has revealed that only a small minority have implemented measures to block artificial intelligence crawlers. Of the sites analyzed, 65.4% serve a robots.txt file, but 64% of these files contain no information about AI crawlers. Thirty-five sites (18.9%) have a hard opt-out, disallowing all AI crawlers with a Disallow: / rule. This hard opt-out is most common among EU media/publishers (66.7%) and Dutch public-interest sites (25.0%). Meanwhile, the majority of government portals and retailers have no AI crawler policy at all. The .well-known/ai-crawler file, which is meant to provide explicit instructions for AI crawlers, has not been widely adopted, with only 20 sites returning a 200 response to the request. This lack of standardization and enforcement leaves a gap in the current system, which the European Commission is actively discussing in its open copyright×AI consultation.",
  "summary": "Dated: 2026-10-01, 17:15–17:22 UTC — single crawl pass, fixed panel, fixed method. Pennyforge research artifact. $0 data (public robots.txt + /.well-known/ai-crawler , plain HTTP with a browser user-agent). Companion data: .tmp-sweep/r141/tdm-census-v2-2026-10-01.json (per-site results, full 12-UA signal table, method). Why this question now Every AI lab says it respects robots.txt for its…",
  "key_points": [
    "185 European websites studied in AI crawler census",
    "65.4% of sites have robots.txt file, but 64% lack AI crawler info",
    "18.9% of sites implement hard opt-out, blocking all AI crawlers"
  ],
  "editors_take": "The findings suggest that while some European websites, particularly media publishers, are taking steps to block AI crawlers, most have not implemented a clear policy, leaving a regulatory gap.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}