Can ChatGPT Read Your Website? How I Found a Block My robots.txt Didn't Show

I investigated if the AI language model ChatGPT could read content from a website. I tested ChatGPT's ability to access the site by asking it to summarize specific pages. When I provided the URLs of my posts, ChatGPT was unable to fetch and summarize the pages, resulting in a 403 error. This error indicates that something on the server level was blocking ChatGPT's access.

I tested the site using the curl command with different user agents, including the one used by ChatGPT. The results showed that while major search engines like Googlebot and Bingbot could access the site, ChatGPT-specific user agents (OAI-SearchBot, ChatGPT-User, and GPTBot) could not. This discrepancy suggested that the block was not due to robots.txt, as it was properly configured and allowed all bots.

The issue turned out to be a server-wide rule maintained by the hosting provider, blocking specific "bad bots" listed by the host. Since shared hosting environments typically do not allow users to modify server-wide rules, the only viable solution was to pass AI crawlers through a proxy service like Cloudflare. By setting up a Cloudflare Worker to handle requests from AI crawlers differently, I was able to allow those requests while maintaining the site's original configuration.

This solution not only resolved the issue with ChatGPT but also provided an additional layer of protection for my website against potential security threats from AI crawlers.