{
  "id": 1633277,
  "title": "The day our scraper fleet shipped nothing — and the afternoon it shipped three",
  "url": "https://urgent.news/2026/08/18/the-day-our-scraper-fleet-shipped-nothing-and-the-afternoon-it",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-18T04:30:46.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/devil_scrapes/the-day-our-scraper-fleet-shipped-nothing-and-the-afternoon-it-shipped-three-4jjo"
  },
  "original_language": "en",
  "account": "On a particular working day last week, nine scrapers were queued, with every release automation functioning perfectly. Despite this, the day concluded with zero listings live. The following afternoon, a remarkable turnaround occurred as three scrapers went from an empty directory to being published within an hour each, each attempt successful without needing retries. The key difference between these two outcomes was not attributed to additional effort, tooling, or engineering skill; it was primarily linked to the target of scraping.\n\nThe report highlights that if the scraping backlog consists of consumer marketplaces and job boards, the throughput is inherently limited by the target's bot-detection budget, not by the scraping velocity itself. To address this issue, they started sorting the backlog by target class instead of revenue. This change involved treating walled targets only after they have undergone a specific tested plan for their defenses.\n\nDuring the day that saw no shipping, nine scrapers targeting consumer marketplaces and job boards queued for publication, all of which were cloud-qualified against datacenter and residential proxies. Consequently, all nine failed, but this failure remained undetected within the process. Each stage worked correctly, the specifications were accurate, tests passed, the pricing was set correctly, and the publisher behaved appropriately. The absence of red flags made the debugging process extremely challenging due to the lack of visible warnings.\n\nHowever, the situation drastically improved in the afternoon when the target class was changed. Three public research and regulatory APIs that do not require an API key were selected for scraping. The process involved scaffolding, implementing, cloud QA, and finally, publishing each scraper within roughly an hour. These public JSON APIs have no incentive to fingerprint or employ CAPTCHA, TLS fingerprint checks, or dynamically changing HTML, making them ideal scraping targets.\n\nOne successful target, OpenAlex, is described as the open replacement for Microsoft Academic Graph, containing information about roughly 250 million scholarly works. It is free, requires no API key, and is well-behaved when interacting with scrapers. To maximize scraping efficiency, the report recommends using cursor paging instead of offset paging, as offset paging becomes ineffective with large corpora. Additionally, identifying yourself with a contact URL in the User-Agent parameter is crucial as it grants access to a faster, more reliable pool of requests.\n\nThe report emphasizes that a QA fixture overly clean, focusing solely on valid data, can lead to delayed failures instead of immediate ones. For example, specifying schema fields as non-nullable types in a QA fixture can mask sparse records, causing validation errors and subsequent failures during data push. To rectify this, the report suggests adding nullability to schema fields, allowing for proper handling of sparse metadata.\n\nFinally, the report advises organizing the scraping backlog by target class first, followed by revenue potential within each class. This approach ensures that keyless public APIs and low-glamour sources are prioritized, resulting in successful scraping attempts. The authors learned this valuable lesson the hard way, acknowledging the importance of target class sorting in achieving efficient and effective scraping outcomes.",
  "summary": "We spent an entire working day last week shipping nothing. Nine scrapers were queued, every piece of release automation worked, and the day ended with zero listings live. That same afternoon, three different scrapers went from empty directory to published in about an hour each, first attempt, no retries. The difference was not effort, tooling, or engineering skill. It was what we pointed at.…",
  "key_points": [
    "Nine scrapers queued on a working day but shipped zero listings",
    "Afternoon turnaround: three scrapers published within an hour each",
    "Target class sorting recommended to improve scraping efficiency"
  ],
  "editors_take": "Shifting the scraping backlog sorting priority from revenue to target class allows for more efficient scraping outcomes by tackling easier targets first, such as public APIs with no bot detection.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}