{
  "id": 9660486,
  "title": "Architectural Breakdown: I Pulled Nine Years of My Own Dev.to Data. The Numbers Were Not What I Expe",
  "url": "https://urgent.news/2026/09/25/architectural-breakdown-i-pulled-nine-years-of-my-own-dev-to-data-the",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-25T00:04:28.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/agenticstack/architectural-breakdown-i-pulled-nine-years-of-my-own-devto-data-the-numbers-were-not-what-i-expe-1e88"
  },
  "original_language": "en",
  "account": "On a chilly Tuesday morning, the author had finished crafting a custom scraper to extract nine years' worth of data from their Dev.to account. Utilizing only Python's standard library, the script bypassed the use of common packages like `requests`, `pandas`, `flask`, and `dotenv`. The dashboard claimed to have produced 847 articles, yet the database contained only 612 records. A concerning 23% of the supposed published content had mysteriously disappeared between 2019 and 2023, becoming soft-deleted by the platform without any notification. This discrepancy of 235 missing articles proved to be the most crucial piece of data collected that weekend.\n\nThe primary issue lay in Dev.to's public API, which had free access but was rate-limited to roughly 30 requests per minute on the anonymous tier. The API lacked any official documentation regarding deleted versus published states, and the pagination mechanism was cursor-based, hidden within response headers. When the author naively fetched page after page and loaded everything into memory, an 8 GB RAM cloud instance became overwhelmed, crashing after just 47 pages. Python's heap memory ballooned to 3.2 GB, triggering the OOM killer and terminating the process.\n\nThe solution was not to add more RAM but to treat the problem as a streaming pipeline rather than a data processing issue. The author built a pipeline utilizing pure Python's standard library, with every component bounded and capacity-limited, and writes batched. The script employed a token bucket rate limiter, exact refill timing, and checksum verification to ensure data integrity.",
  "summary": "![ Architecture Diagram ]( https://image.pollinations.ai/prompt/high+performance+cloud+systems+I+Pulled+Nine+Years+of+My+Own++round+2?width=800&height=400&nologo=true ) # I Pulled Nine Years of My Own Dev.to Data. The Numbers Were Not What I Expected. It was 2:47 AM on a Tuesday when I finished a custom scraper for nine years of my own Dev.to history. I bypassed every bloated npm package using…",
  "key_points": [
    "Author scraped nine years of Dev.to data using Python's standard library",
    "847 articles claimed by dashboard, but database only contained 612 records",
    "23% of supposed published content mysteriously disappeared between 2019-2023"
  ],
  "editors_take": "The author's experience reveals a significant blind spot in Dev.to's public API, where a lack of clear documentation and rate limits can lead to data loss and processing failures for users relying on it.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}