I Analyzed 2,849 Crawler Requests. Here's What Search Bots Actually Do on New Sites.

An SEO analyst examined 2,849 crawler requests to a single website over the course of one day. The results suggest that more than half of the bot traffic comes from AI companies training models, rather than search engines indexing pages. The top content visited included tutorial pages, which received 50 combined crawler visits, compared to blog posts with ~18 and individual tools with ~20 visits.

This is likely because tutorials are text-heavy, semantically dense, and link-rich, making them easy for bots to parse and index. Googlebot, the most aggressive crawler, made 80 requests, which included checking the robots.txt file, loading the sitemap, following links to tutorials, visiting blog posts, and sporadically checking tool pages.

The findings highlight the importance of having a properly configured robots.txt file, submitting sitemaps, writing tutorials instead of just tools, internal linking between tutorials and tools, and utilizing IndexNow for Bing verification. Additionally, AI bots are actively crawling sites, ingesting content for the next generation of AI models.

Googlebot's aggressive crawling demonstrates that indexing is separate from merely visiting a site, with new sites typically experiencing a 1-4 week lag before their content is indexed.