{
  "id": 12523572,
  "title": "I Analyzed 2,849 Crawler Requests. Here's What Search Bots Actually Do on New Sites.",
  "url": "https://urgent.news/2026/10/07/i-analyzed-2-849-crawler-requests-heres-what-search-bots-actually-do",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-07T02:25:25.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/mou1z/i-analyzed-2849-crawler-requests-heres-what-search-bots-actually-do-on-new-sites-31i"
  },
  "original_language": "en",
  "account": "An SEO analyst examined 2,849 crawler requests to a single website over the course of one day. The results suggest that more than half of the bot traffic comes from AI companies training models, rather than search engines indexing pages. The top content visited included tutorial pages, which received 50 combined crawler visits, compared to blog posts with ~18 and individual tools with ~20 visits. This is likely because tutorials are text-heavy, semantically dense, and link-rich, making them easy for bots to parse and index. Googlebot, the most aggressive crawler, made 80 requests, which included checking the robots.txt file, loading the sitemap, following links to tutorials, visiting blog posts, and sporadically checking tool pages. The findings highlight the importance of having a properly configured robots.txt file, submitting sitemaps, writing tutorials instead of just tools, internal linking between tutorials and tools, and utilizing IndexNow for Bing verification. Additionally, AI bots are actively crawling sites, ingesting content for the next generation of AI models. Googlebot's aggressive crawling demonstrates that indexing is separate from merely visiting a site, with new sites typically experiencing a 1-4 week lag before their content is indexed.",
  "summary": "I Analyzed 2,849 Crawler Requests. Here's What Search Bots Actually Do on New Sites. I run k1r4.space — a site with 30 HTML tools, a 31-post blog, and 4 SEO tutorials. I logged every single request for a full day. Here's what 321 crawler requests from 12 different bots told me about SEO for new sites. The Setup My site runs on a single VPS (Debian 13, AMD Ryzen 5 3600, 62GB RAM, no GPU) behind…",
  "key_points": [
    "Over 2,849 crawler requests analyzed in one day",
    "AI companies dominate bot traffic, not search engines",
    "Tutorials receive most crawler visits due to text density"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}