{
  "id": 14931181,
  "title": "How We Stopped a Weekly Batch Job From Taking Down a Live MySQL Service",
  "url": "https://urgent.news/2026/10/16/how-we-stopped-a-weekly-batch-job-from-taking-down-a-live-mysql",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-16T05:33:40.000Z",
  "source": {
    "name": "HackerNoon",
    "slug": "hackernoon",
    "url": "https://hackernoon.com/how-we-stopped-a-weekly-batch-job-from-taking-down-a-live-mysql-service?source=rss"
  },
  "original_language": "en",
  "account": "The incident began with a series of gateway timeouts on a service that seemed unaltered. No recent updates, no unusual traffic, and no apparent log clues. The callers experienced timeouts, but outsiders might have attributed it to a slow UPDATE running in the background. The root cause was a MySQL table performing dual functions: serving live inserts continuously and handling a massive weekly bulk update. This single UPDATE held row locks for its entire duration, causing new inserts to queue up and eventually time out. While the problem seemed straightforward on paper, it had caused multiple outages in production. This scenario is more common than one might think, often arising when a single data store is expected to excel at two vastly different tasks. The service in question is an internal tool that processes deletion requests. Teams map columns to labels and submit deletion requests via these labels. Unlike straightforward Parquet or JSON tables, many assets lack Delete DML support, necessitating a centralized deletion service. This service handles tens of thousands of requests daily, ranging from compliance deletes to data cleanup and test data removal. When the service was initially designed, it handled a small number of deletion requests per day, and the schema was adequate. However, as teams onboarded more assets, the volume grew beyond what the schema could handle. It took weeks for the traffic to reach a point where the schema's limitations became apparent, and redesigning the table was not a simple solution. The weekly bulk update ran against the primary table, holding row locks for the entire duration and causing new inserts to queue up and time out. The real issue was not the deletion itself but the subsequent bookkeeping required after the job completed. The fix involved splitting the table: new requests went into a separate queue table, while live writes and status checks accessed the primary table. This approach preserved a single source of truth while alleviating the strain on the primary table during the weekly bulk update.",
  "summary": "How a team eliminated production timeouts caused by MySQL bulk updates using queue tables, resumable rotation, and workload isolation.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}