{
  "id": 193662,
  "title": "A page my crawler could not read is not a page that does not exist. Six times I got that wrong, twice while writing this",
  "url": "https://urgent.news/2026/08/05/a-page-my-crawler-could-not-read-is-not-a-page-that-does-not-exist",
  "topic": "culture",
  "section": "Culture",
  "published": "2026-08-05T22:08:12.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/blueticks/a-page-my-crawler-could-not-read-is-not-a-page-that-does-not-exist-six-times-i-got-that-wrong-4pb5"
  },
  "original_language": "en",
  "account": "This story recounts the author's experiences with their measurement rig, a set of scripts designed to monitor their web pages. Despite the rig's usefulness, it has failed in specific ways six times, with each instance resulting from the same mistake. The author notes that each failure was recorded as a fact about the world, such as a directory serving an anti-bot challenge instead of its content, causing the rig to incorrectly identify a page as blocked. In another instance, a listing was marked as not live for six days when it was actually live, leading to incorrect counts of live pages and influencing the decision on whether a directory was worth the effort. The author also discusses how a tool's limitations can influence their understanding of reality, such as a directory appearing unreadable due to a command line HTTP client's inability to render client-side content. Another example involves an outbound link to the author's domain, which the crawler failed to find after the page took longer than eleven seconds to finish rendering. The author concludes by establishing a rule for their future measurements: a positive result from a crawler is evidence, while a negative result is a hypothesis.",
  "summary": "I have been running a small measurement rig for two weeks: a set of scripts that load our pages in a fresh anonymous browser context and record what is actually there. Directory listings, articles, search positions. It has been genuinely useful and it has been wrong in one specific way, six separate times, and the six are the same mistake. Every time, a script failed to read something, and I…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}