AI crawler user agents are self-reported: 468 real fetches, 991 fake ones
I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up. Then I re-cut eight days of logs by verification result. Requests that actually fetched an article: 468. Requests probing for .env and friends: 991. Of everything calling itself GPTBot, Cloudflare…
The article explores the issue of AI crawler traffic on websites, highlighting the discrepancy between claimed User-Agent strings and actual verified bot activity. By analyzing eight days of logs, the author found that only 468 requests actually fetched an article, while 991 requests were probing for sensitive files like .env, .git, and secrets.
Out of 547 requests presenting as ChatGPT-User, only 211 requests could be traced back to OpenAI. The remaining requests were unverified impersonators. Similarly, 455 Perplexity-branded requests were also unverified. The author emphasizes that a claim in the User-Agent string does not equate to actual activity, and verification is crucial.
The article outlines the process of separating impersonators on a Cloudflare free plan, using two filters: verifiedBotCategory and whether the requested path exists in the sitemap. Verified bots fetching real pages are the only numbers worth reporting. The author provides a query to extract the data, with constraints such as limiting requests to one day per request and a short retention period.
The article also discusses specific cases like Google-Extended, which does not have a separate HTTP user agent string and is not identified by the Google-Extended UA. Instead, it states that Google crawling occurs under the existing Google user agents. The author concludes by reminding readers that verification is the only reliable way to separate fact from fakery in AI crawler traffic.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
