{
  "id": 5880526,
  "title": "984 Requests Said They Were Perplexity. None Could Prove It.",
  "url": "https://urgent.news/2026/09/06/984-requests-said-they-were-perplexity-none-could-prove-it",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-06T01:32:13.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/roadleon/984-requests-said-they-were-perplexity-none-could-prove-it-33nm"
  },
  "original_language": "en",
  "account": "984 requests arrived at a website claiming to be various crawlers, but none of them could be verified as legitimately sent by those entities. A user agent string is not concrete proof of identity, as anyone can create one. Verification is only possible when a vendor publishes the IP addresses their crawlers use. Some vendors provide such lists, while others do not. In this case, Meta, ByteDance, and Amazon do not publish any IP ranges or reverse DNS schemes, making verification impossible for their crawlers. For Anthropic and Common Crawl, the verification lists were outdated. Anthropic began publishing a range list on August 18 but I was still using a snapshot from July 20. Common Crawl published a list on August 11, but it did not get updated in the thirty-day analysis period. The Perplexity requests were verified against a list that was last updated in October 2025, showing that the list was incomplete and not current. This highlights the importance of having fresh and accurate verification lists to accurately identify the crawlers sending requests.",
  "summary": "This morning I told my own website that I was ClaudeBot. It took one line of curl and a user agent string copied out of Anthropic's own documentation. Three requests to one article page. Then I opened my dashboard: Anthropic, ClaudeBot, training — 1,698 requests had become 1,701, last seen at 09:22. None of them was ClaudeBot. They came from a laptop in Japan, over ordinary home broadband, from…",
  "key_points": [
    "984 requests claimed to be crawlers",
    "Verification impossible without IP ranges",
    "Perplexity requests from outdated lists"
  ],
  "editors_take": "The inability to verify Perplexity requests due to outdated lists highlights the importance of having fresh and accurate verification lists to accurately identify crawlers sending requests.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}