{
  "id": 18392828,
  "title": "I checked what 31 big websites tell AI crawlers in robots.txt, and found a bug in my own tool",
  "url": "https://urgent.news/2026/11/05/i-checked-what-31-big-websites-tell-ai-crawlers-in-robots-txt-and",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-05T22:59:19.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/alorenzom/i-checked-what-31-big-websites-tell-ai-crawlers-in-robotstxt-and-found-a-bug-in-my-own-tool-20o6"
  },
  "original_language": "en",
  "account": "A study examining the robots.txt files of 40 major websites revealed differences in how they allow or block AI crawlers. Of the 31 sites that could be audited, 10 blocked at least one AI search or assistant bot on their home page. The remaining 18 sites did not block any of the 12 types of crawlers examined.\n\nThe 10 sites that blocked at least one search or assistant bot included The Guardian, BBC, Reuters, The Verge, TechCrunch, Wired, Amazon, eBay, Reddit, and IMDb. Notably, Reuters, Reddit, and IMDb have general User-agent: * rules that close the home page to any bot without its own specific rules, which does not specifically target AI crawlers.\n\nThe analysis also found a bug in the researcher's own tool, PageSignal, an AI visibility API. During the initial run, Figma was incorrectly reported as blocking AI search bots. Upon investigation, it was discovered that Python's built-in urllib.robotparser library did not correctly interpret certain directives in the robots.txt file, such as the use of '$' as an end-of-URL marker and the handling of bot names containing other bot names.\n\nCorrecting this bug resolved the issue, bringing the tool's results in line with the actual behavior of the websites being audited. The researcher suggests that while robots.txt serves as a request, not a lock, it can provide valuable insights into a website's stance on AI crawlers. However, it's important to remember that these files can change over time, and being readable does not guarantee citation.",
  "summary": "Can ChatGPT, Claude or Perplexity read your website? The first place to look is a small text file: robots.txt . I wanted to see what big websites actually say in it, so I picked 40 well-known sites and audited their home pages for 12 AI crawlers. The results split much more cleanly by industry than I expected. The study also found a bug in my own tool. More on that below, because it's a mistake…",
  "key_points": [
    "10 out of 31 major sites blocked AI crawlers on home pages",
    "Reuters, Reddit, and IMDb use general User-agent:  rules",
    "Bug in researcher's tool misreported Figma's AI crawler stance"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}