I checked what 31 big websites tell AI crawlers in robots.txt, and found a bug in my own tool
Can ChatGPT, Claude or Perplexity read your website? The first place to look is a small text file: robots.txt . I wanted to see what big websites actually say in it, so I picked 40 well-known sites and audited their home pages for 12 AI crawlers. The results split much more cleanly by industry than I expected. The study also found a bug in my own tool. More on that below, because it's a mistake…
A study examining the robots.txt files of 40 major websites revealed differences in how they allow or block AI crawlers. Of the 31 sites that could be audited, 10 blocked at least one AI search or assistant bot on their home page. The remaining 18 sites did not block any of the 12 types of crawlers examined.
The 10 sites that blocked at least one search or assistant bot included The Guardian, BBC, Reuters, The Verge, TechCrunch, Wired, Amazon, eBay, Reddit, and IMDb. Notably, Reuters, Reddit, and IMDb have general User-agent: * rules that close the home page to any bot without its own specific rules, which does not specifically target AI crawlers.
The analysis also found a bug in the researcher's own tool, PageSignal, an AI visibility API. During the initial run, Figma was incorrectly reported as blocking AI search bots. Upon investigation, it was discovered that Python's built-in urllib.robotparser library did not correctly interpret certain directives in the robots.txt file, such as the use of '$' as an end-of-URL marker and the handling of bot names containing other bot names.
Correcting this bug resolved the issue, bringing the tool's results in line with the actual behavior of the websites being audited. The researcher suggests that while robots.txt serves as a request, not a lock, it can provide valuable insights into a website's stance on AI crawlers. However, it's important to remember that these files can change over time, and being readable does not guarantee citation.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
