Our lock expired while the first worker was still writing invoices
On the second of June two hundred and twelve of our business customers received their May invoice twice. Nine of them received it three times. Invoicing runs on eight workers that take customer accounts from a list. To stop two workers invoicing the same customer, each takes a lock in Redis first: a SET with NX and an expiry of sixty seconds, keyed on the customer. A minute was generous, because…
On June 2nd, 212 of our business clients received their May invoices twice, while nine experienced three duplicates. The invoicing process relies on eight workers who manage customer accounts from a list. To prevent two workers from invoicing the same customer, each worker initially acquires a lock in Redis with a sixty-second expiration, keyed on the customer. This lock is released when the worker completes the invoice.
May was a busy month due to a higher number of usage lines and slow tax service responses. Consequently, the largest invoices took between two and five minutes to complete. After the sixty-second lock expiry, Redis remained unaware if the key holder was still working. Another worker detected the absence of a lock, took possession, and started the invoice from scratch, causing the first worker's work to be duplicated.
Once the first worker finished, saved the invoice, and emailed it, they deleted the key, which by then belonged to the second worker. This scenario allowed a third worker to commence the process. The issue arose when a lock with an expiry acts as a lease, expiring at a specific time regardless of whether the work is finished, leaving the holder unaware of the expiry.
Our invoicing system enforces a unique constraint on customer and billing period, ensuring that an email about the invoice is only sent after a successful database insert. Thus, a second worker encounters an error during the database insertion rather than at the customer's inbox. The lock persists to prevent wasted effort, and each acquisition includes a fencing token, a continuously incrementing number checked by the database on every write.
Workers maintain the lock for twenty seconds while actively working and abandon the job if the renewal fails. We also receive alerts for any lock that expires while its holder is still running. Every lock expiry is an estimate of the slowest possible job duration. Our estimation proved accurate for four years until the workload grew beyond our expectations.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
