{
  "id": 18912688,
  "title": "How an autosave feature took down production — and five fixes that stuck",
  "url": "https://urgent.news/2026/11/08/how-an-autosave-feature-took-down-production-and-five-fixes-that-stuck",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-08T10:26:04.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/noamtzuberi/how-an-autosave-feature-took-down-production-and-five-fixes-that-stuck-mmj"
  },
  "original_language": "en",
  "account": "An editing tool that automatically saves user progress was the cause of a production outage. The issue was not a single bug, but a combination of five seemingly reasonable choices that added up. Here's how the situation unfolded and the fixes implemented.\n\n1. The editor sent the cursor position and scene data to the server on a timer. The server wrote these details to the database every time, even when nothing had changed. The baseline on the client never advanced, so each \"save\" looked like a new change. Under normal traffic, this was undetectable. However, when usage increased, the database spent most of its time writing identical rows.\n\n2. To address the issue, the client kept a baseline and sent only real changes, while the server skipped writes where the value was unchanged. A no-op save should not consume any resources.\n\n3. Each save first selected the row from the database, then updated it. This resulted in two round trips, where one conditional update would suffice. By doubling the load where it was most needed, this approach slowed down the feature.\n\n4. An asynchronous database path used a synchronous driver call within an async server. During high load, this blocking call slowed down the event loop, causing requests that didn't involve the database — such as health checks — to queue behind it.\n\n5. Health checks, which were stuck behind the slow database, led Kubernetes to restart pods. Fresh pods reconnected and retried pending saves, exacerbating the problem. The restarts worsened the outage.\n\nTo resolve these issues, the following fixes were implemented:\n\n1. The client compared values before writing to the server, sending only real changes.\n2. A single upsert with a condition was used instead of two round trips for updates.\n3. Synchronous I/O was prohibited on the event loop, enforced by a lint rule.\n4. Health checks and readiness probes were separated. Liveness checks only verified if the process was alive, without touching dependencies. Readiness checks could examine the database and mark the pod as unhealthy without terminating it.\n5. The autoscaler was configured to monitor request latency and queue depth, as well as to maintain headroom in quota, and alerts were set up for error rate and latency.",
  "summary": "An editor I built saves your work as you type. That's the feature users love most, and one evening it was the reason the whole app stopped answering. No single bug caused it. Five ordinary decisions, each reasonable on its own, lined up. Here they are, with what I changed. 1. Saving things that hadn't changed The editor sent the cursor position and the current scene on a timer. The server wrote…",
  "key_points": [
    "Autosave feature caused production outage by sending cursor and scene data to server on timer",
    "Client kept baseline and sent only real changes, server skipped writes where value was unchanged"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}