{
  "id": 15978795,
  "title": "Which AI bots visit my website? I counted for 30 days",
  "url": "https://urgent.news/2026/10/22/which-ai-bots-visit-my-website-i-counted-for-30-days",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-22T00:56:17.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/yappatyih/which-ai-bots-visit-my-website-i-counted-for-30-days-2cnj"
  },
  "original_language": "en",
  "account": "In a 30-day period from October 1st to October 20th, 2026, Cloudflare counted around 4,040 requests from AI crawlers to the site. Among the top visitors were Google's Googlebot, Anthropic's ClaudeBot, OpenAI's GPTBot, Apple's Applebot, and Microsoft's BingBot. Only 26 out of the successful requests were made by AI assistants fetching pages because a person asked for information. The majority of the requests were for training AI models (about 1,720) and building search indices (about 1,190). The site's sitemaps were read 687 times, and none of the posts were read more than 66 times. Notably, no AI assistant cited the site in its responses. The most frequently accessed file was the sitemap-posts.xml list, followed by the sitemap-index.xml and the home page. Scanning for passwords and keys was also a common activity among the AI crawlers, though these requests resulted in 404 errors as those files did not exist on the site.",
  "summary": "First published on yappatyih.com . TL;DR AI crawler detection means finding out which bots that collect pages for AI companies visit your website, how often, and what they read, usually from the name each bot gives in its requests. In the 30 days to 20 October 2026, Cloudflare counted about 4,040 requests to this site from the AI crawlers it recognises. Google sent the most, then Anthropic,…",
  "key_points": [
    "Cloudflare tracked 4,040 AI crawler requests in 30 days",
    "Googlebot, ClaudeBot, GPTBot, Applebot, BingBot were top visitors",
    "1,720 requests were for AI model training, 1,190 for search indexing"
  ],
  "editors_take": "The overwhelming majority of AI crawler traffic prioritizes model training and search indexing over direct user assistance, signaling that web content serves as raw data rather than a source for citations.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}