Cloudflare's AI block names eight crawlers. None is ChatGPT's search bot
Eight user agents, and the one that decides whether ChatGPT cites you is not among them. An r/SEO post from April, 53 points and 40 comments, says Cloudflare quietly cut the author's site off from ChatGPT, from Perplexity and from Google's AI Overviews. I went and read the block. It names ChatGPT's training crawler and not its search crawler, and it never mentions Perplexity at all. The Google…
The recent changes to Cloudflare's managed robots.txt settings have inadvertently blocked certain AI crawlers while allowing others. This includes barring ChatGPT's training crawler (GPTBot) but permitting its search crawler (Google-Extended). The block also impacts Perplexity's crawler and Google's Gemini model training crawler (Google-Extended). The block's selective nature appears deliberate, affecting crawlers based on whether they are designated as "AI crawlers" or "AI search" crawlers in Cloudflare's documentation.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
