A robots.txt that allows GPTBot does not mean GPTBot can reach your site
I built SeenLens, a one-time AI readiness audit for websites. I wanted to know what happens when a page's published crawler rules and its HTTP responses tell different stories. I scanned one public URL on each of 30 sites. In that run, 5 returned HTTP 403 or 202 to the scanner. I classified 4 as bot-wall responses based on their headers and body size: a challenge header on one response, bot…
Just because a site includes a robots.txt file that allows the GPTBot crawler doesn't automatically mean GPTBot can access the site's content. The server's response to a request for a specific page can conflict with the directives in robots.txt. A content delivery network (CDN) or web application firewall (WAF) may return an error, rate-limit the request, or present a challenge page instead of serving the full page.
These challenges can prevent GPTBot from fully accessing the site, even if the crawlers' tokens are technically allowed by robots.txt.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
