{
  "id": 14132875,
  "title": "Our lock expired while the first worker was still writing invoices",
  "url": "https://urgent.news/2026/10/13/our-lock-expired-while-the-first-worker-was-still-writing-invoices",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-13T06:02:27.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sergey_shinder_ab2d943365/our-lock-expired-while-the-first-worker-was-still-writing-invoices-5371"
  },
  "original_language": "en",
  "account": "On June 2nd, 212 of our business clients received their May invoices twice, while nine experienced three duplicates. The invoicing process relies on eight workers who manage customer accounts from a list. To prevent two workers from invoicing the same customer, each worker initially acquires a lock in Redis with a sixty-second expiration, keyed on the customer. This lock is released when the worker completes the invoice.\n\nMay was a busy month due to a higher number of usage lines and slow tax service responses. Consequently, the largest invoices took between two and five minutes to complete. After the sixty-second lock expiry, Redis remained unaware if the key holder was still working. Another worker detected the absence of a lock, took possession, and started the invoice from scratch, causing the first worker's work to be duplicated.\n\nOnce the first worker finished, saved the invoice, and emailed it, they deleted the key, which by then belonged to the second worker. This scenario allowed a third worker to commence the process. The issue arose when a lock with an expiry acts as a lease, expiring at a specific time regardless of whether the work is finished, leaving the holder unaware of the expiry.\n\nOur invoicing system enforces a unique constraint on customer and billing period, ensuring that an email about the invoice is only sent after a successful database insert. Thus, a second worker encounters an error during the database insertion rather than at the customer's inbox. The lock persists to prevent wasted effort, and each acquisition includes a fencing token, a continuously incrementing number checked by the database on every write.\n\nWorkers maintain the lock for twenty seconds while actively working and abandon the job if the renewal fails. We also receive alerts for any lock that expires while its holder is still running. Every lock expiry is an estimate of the slowest possible job duration. Our estimation proved accurate for four years until the workload grew beyond our expectations.",
  "summary": "On the second of June two hundred and twelve of our business customers received their May invoice twice. Nine of them received it three times. Invoicing runs on eight workers that take customer accounts from a list. To stop two workers invoicing the same customer, each takes a lock in Redis first: a SET with NX and an expiry of sixty seconds, keyed on the customer. A minute was generous, because…",
  "key_points": [
    "212 clients received duplicate May invoices",
    "Locks expired despite workers still invoicing",
    "Issue resolved with fencing token and expiry alerts"
  ],
  "editors_take": "Static lock expiration windows fail to account for unpredictable workload surges, turning a previously reliable concurrency control mechanism into a source of duplicate billing errors.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}