Which AI bots visit my website? I counted for 30 days
First published on yappatyih.com . TL;DR AI crawler detection means finding out which bots that collect pages for AI companies visit your website, how often, and what they read, usually from the name each bot gives in its requests. In the 30 days to 20 October 2026, Cloudflare counted about 4,040 requests to this site from the AI crawlers it recognises. Google sent the most, then Anthropic,…
In a 30-day period from October 1st to October 20th, 2026, Cloudflare counted around 4,040 requests from AI crawlers to the site. Among the top visitors were Google's Googlebot, Anthropic's ClaudeBot, OpenAI's GPTBot, Apple's Applebot, and Microsoft's BingBot. Only 26 out of the successful requests were made by AI assistants fetching pages because a person asked for information.
The majority of the requests were for training AI models (about 1,720) and building search indices (about 1,190). The site's sitemaps were read 687 times, and none of the posts were read more than 66 times. Notably, no AI assistant cited the site in its responses. The most frequently accessed file was the sitemap-posts.xml list, followed by the sitemap-index.xml and the home page.
Scanning for passwords and keys was also a common activity among the AI crawlers, though these requests resulted in 404 errors as those files did not exist on the site.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
