Who's actually blocking AI crawlers in Europe? A 185-site census (first dated point)
Dated: 2026-10-01, 17:15–17:22 UTC — single crawl pass, fixed panel, fixed method. Pennyforge research artifact. $0 data (public robots.txt + /.well-known/ai-crawler , plain HTTP with a browser user-agent). Companion data: .tmp-sweep/r141/tdm-census-v2-2026-10-01.json (per-site results, full 12-UA signal table, method). Why this question now Every AI lab says it respects robots.txt for its…
A census of 185 European websites has revealed that only a small minority have implemented measures to block artificial intelligence crawlers. Of the sites analyzed, 65.4% serve a robots.txt file, but 64% of these files contain no information about AI crawlers. Thirty-five sites (18.9%) have a hard opt-out, disallowing all AI crawlers with a Disallow: / rule.
This hard opt-out is most common among EU media/publishers (66.7%) and Dutch public-interest sites (25.0%). Meanwhile, the majority of government portals and retailers have no AI crawler policy at all. The .well-known/ai-crawler file, which is meant to provide explicit instructions for AI crawlers, has not been widely adopted, with only 20 sites returning a 200 response to the request.
This lack of standardization and enforcement leaves a gap in the current system, which the European Commission is actively discussing in its open copyright×AI consultation.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
