{
  "id": 18235857,
  "title": "What 4,700 bot visits taught me about AI crawlers (and a tool to check any site's robots.txt",
  "url": "https://urgent.news/2026/11/05/what-4-700-bot-visits-taught-me-about-ai-crawlers-and-a-tool-to-check",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-05T03:03:19.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/bal02/what-4700-bot-visits-taught-me-about-ai-crawlers-and-a-tool-to-check-any-sites-robotstxt-2549"
  },
  "original_language": "en",
  "account": "I recently wanted to determine how many visitors on my small side projects were genuine humans. The results revealed that most traffic came from AI crawlers, with no reliable way to identify which ones, how often, or where they originated. This led me to create BotSpotter, a tool that provides a live map of AI crawlers visiting participating sites, and is open source under the MIT license. Here are four key insights into AI crawlers based on the first week of data:\n\n1. Approximately one in every 18 big-name bots is falsifying its identity. Major companies like OpenAI, Google, Microsoft, Apple, Perplexity, DuckDuckGo, and Common Crawl publish the IP ranges used by their crawlers. BotSpotter cross-references hits claiming to be these bots against the published ranges before discarding the IP. After 1,500 hits, 1,425 were verified, while 75 were found to be impostors. The most common disguise is Googlebot (40 of the 75), followed by GPTBot (20), Bingbot (12), and PerplexityBot (3). If you rate-limit or whitelist by User-Agent, you may inadvertently admit this traffic.\n\n2. The majority of AI crawling remains anonymous. Out of 4,770 bot visits, 2,494 (56%) came from bots without identifiable origins. These bots are identified through fingerprinting based on UA family, header order, and network, meaning the same bot retains the same name. While there are identifiable AI companies, the majority of machine traffic does not disclose its identity.\n\n3. New domains attract a surge of bot visits, which then gradually decrease. Upon launching, I received over 500 bot hits per day, which dropped to around 320 after two weeks. This spike occurred as every search and SEO crawler performed a discovery pass on the new domain (Googlebot, Bingbot, Ahrefs, Semrush, Meta, etc.). If you're tracking bot traffic on a freshly created site, the initial week of data isn't representative of your typical traffic.\n\n4. The geographical distribution of bot hosts reveals interesting patterns. When hosting bots, Google Cloud was the most common (2,117 hits), followed by AWS (293), Hetzner (287), OVH (270), Azure (170), Apple (165), Meta (85), and DigitalOcean (77). Hetzner and OVH being among the top hosts indicate that a significant number of scrapers and unidentified crawlers rent inexpensive European virtual machines. By checking any site's robots.txt against AI crawlers, you can gain insights into how well protected your site is from these bots. The most useful feature of BotSpotter is its ability to evaluate a site's robots.txt against 16 AI crawler tokens, providing a grade per crawler indicating whether the traffic is allowed, restricted to certain paths, or completely blocked.",
  "summary": "A few weeks ago I wanted to answer a simple question: how many of the \"visitors\" on my small side projects are actually people? The answer turned out to be: not many. Most of the traffic was crawlers, a lot of them from AI companies, and I had no easy way to see which ones, how often, or where they came from. Server logs tell you a User-Agent string and an IP, and the User-Agent can say anything…",
  "key_points": [
    "Approximately one in every 18 big-name bots is falsifying its identity",
    "The majority of AI crawling remains anonymous"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}