I tracked AI crawlers on my new GEO site for 30 days — here's what I learned

About two months ago, I launched a website dedicated to Generative Engine Optimization (GEO). Instead of just monitoring Google rankings, I installed Cloudflare D1 logging to track every AI crawler visit. This 30-day log reveals several key findings.

The most frequent crawlers are Bingbot, averaging 98 visits per day, with ClaudeBot following at 18-22 hits daily. GPTBot and PerplexityBot bring in 8-12 and 3-5 hits per day respectively, though more sporadically. The standout is Bytespider, the ByteDance owned crawler, which consistently scans a site 500+ times daily. This likely represents massive data collection.

Several insights emerge. Bing is now the leading platform for AI crawlers, outpacing Google. Bing Webmaster Tools becomes crucial for GEO optimization. ClaudeBot's crawling pattern is remarkably consistent, suggesting a scheduled indexing rather than a random scan. Expect a 24-48 hour delay before Claude sees your content after publication.

Bytespider's aggressive scanning is notable for smaller sites, indicating its role in data collection. External referrers carry more weight than anticipated. A single Reddit post from an unknown domain (sahammurah.com) triggered 8 AI crawler visits. This demonstrates that AI bots discover content through external mentions.

Based on this data, I've adjusted my strategy. I added an llms.txt file, a robots.txt equivalent for AI crawlers. All articles now include FAQPage and Article Schema markup, with answer-first paragraphs. Each piece cites verifiable data sources. I'm curious to hear from others tracking AI crawlers. What patterns have you observed?