48-Hour Field Notes: A Crash Mid-Batch Proved My Queue Had No Memory
My batch job had a very confident opinion about progress, and that opinion lived entirely in a Python variable. If the process died, the variable died with it, and the next run started from zero like nothing had ever happened. Sound familiar? I spent a 48-hour window on this instead of shipping the feature I actually wanted to ship. Here is the thing that annoyed me most: my earlier fix was…
In the midst of a crash during a batch job, an issue emerged where the queue had no memory of previously claimed items that had not yet been completed. The Python variable responsible for tracking progress died along with the process, causing subsequent runs to start from scratch. Inspecting the code, it became clear that the earlier fix, which involved idempotency-hashing inputs, did not address the problem of missing items due to crashes.
To resolve this, a journal was created to survive sudden terminations and keep track of which items were claimed, completed, or stranded.
The challenge was to handle the window between claiming an item and committing the outcome. This gap was where queues lost data, and it was where the first version of the code had lost a pending item permanently. To tackle this, a lease system was implemented, with a monotone clock check and the rule that a claim expires after a certain period.
By introducing four states (pending → claimed → done | failed) and carrying lease information in the claimed rows, the issue of unrecoverable states was transformed into a recoverable one.
The solution was implemented in a runner.py script using SQLite on purpose for its simplicity - a single file that can be easily copied and inspected. The script includes functions for connecting to the database, initializing the work and ledger tables, claiming the next item, and finishing the job with an outcome. By enforcing a lease expiration, the system ensures that no data is lost even in the event of a crash, allowing workers to safely pick up items and making the entire process more robust and reliable.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
