{
  "id": 5359892,
  "title": "Crawling at Scale: Scheduling, Deduplication, and Storage",
  "url": "https://urgent.news/2026/09/03/crawling-at-scale-scheduling-deduplication-and-storage",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-03T16:01:48.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/3ni8ma/crawling-at-scale-scheduling-deduplication-and-storage-3e5p"
  },
  "original_language": "en",
  "account": "Frontier Design: What to Fetch Next\nThe priority queue of URLs for the crawl frontier includes rules such as staying within allowed hosts/paths, re-fetching pages based on freshness, and prioritizing document pages over index pages. A frontier without priorities can lead to unnecessary repetition of the same content. Respecting politeness by adding delays between requests, retrying with exponential backoff on error codes like 429 and 5xx, and using conditional fetches with Last-Modified and ETag headers are essential. Failure to do so may result in being blocked, negatively impacting other crawlers as well.",
  "summary": "The Frontier: What to Fetch Next The crawl frontier is a priority queue of URLs, not a set. Rules that matter: scope filtering (stay on allowed hosts/paths), freshness (when to re-fetch a page), and prioritization (document pages over index pages). A frontier without priorities degenerates into hammering the same section. Politeness as a Hard Requirement The politeness policy: delay between…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}