I measured 528 sites' robots.txt: two thirds blocking GPTBot also block the crawler that cites them
Two thirds of the sites blocking GPTBot are also blocking the crawler that sends them traffic. I measured it. Blocking GPTBot is a reasonable decision. OpenAI documents it as the crawler that gathers training data, you get nothing back for it, and plenty of publishers have decided that trade isn't worth making. Fine. The problem is that GPTBot is not the only OpenAI token in your robots.txt , and…
A study found that two thirds of the sites blocking GPTBot were also blocking other OpenAI crawlers. Measuring 528 sites, it was discovered that 66.5% of those blocking GPTBot were also blocking traffic-generating crawlers. Only 10.2% of the sites blocked training crawlers but allowed discovery crawlers. Just 17.2% blocked one traffic-generating crawler while allowing Googlebot and others.
This data shows a deliberate decision by some markets to block both types of crawlers while others, like the Netherlands, block both at higher rates. The issue lies in the complexity of robots.txt rules, which can unintentionally block both types of crawlers due to how they interact. A tool has been created to help identify these errors, making it easier to understand and adjust the settings.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.