Two thirds of sites blocking GPTBot are also blocking the crawler that brings them traffic

A recent analysis reveals a striking trend among major websites: nearly two-thirds of the top 1 million domains block GPTBot, the AI crawler used by OpenAI to train its models. However, an even more concerning finding is that these same websites block OAI-SearchBot, the crawler responsible for building the index behind ChatGPT's answers and enabling live answer fetches.

This double blocking of AI crawlers has significant implications for website traffic and discoverability. While blocking GPTBot results in opting out of model training, blocking OAI-SearchBot removes the site from the platform that brings in substantial visitor traffic. Curiously, 66.5% of the 161 sites blocking GPTBot also block OAI-SearchBot, despite the fact that only 10.2% of all sites choose to block only the training crawler while allowing discovery.

This discrepancy suggests that many high-ranking websites are inadvertently opting out of the growing segment of search traffic powered by AI. The study also highlights that 17.2% of sites block an AI crawler that brings in traffic while still allowing Googlebot. This suggests that these websites are not opting out of search results but specifically avoiding the half of search that has grown in popularity.

The research was conducted using a strict RFC 9309 matcher on the Tranco top-1M list, focusing on robots.txt files of the highest-ranked domains across seven major markets. The findings demonstrate the importance of accurately interpreting and implementing robots.txt rules to maintain optimal visibility and search traffic.