48-Hour Field Notes: My Batch Job Trusted Local Disk, and a Restart Re-Bought Every Token

In the first six hours, the batch job processed five documents serially, writing summaries into an output table. The process was straightforward: claim an item, build a prompt, call the model, write the result, and mark the item as finished. Despite its simplicity, the author trusted the smoke test because it was fast and gave green output, but this was an unreliable proxy for proper recovery behavior.

The issue arose when the worker restarted mid-run, causing duplicate processing of previously completed items. This happened because the resume mechanism depended on the worker's local disk, which does not guarantee durability if the worker restarts. The author realized that they had been making three assumptions without testing them: the local disk survives a restart long enough for a checkpoint file to matter, the job already ran can be figured out from its own state, and a restart is similar to a crash, so the same recovery logic applies to both.

To address this, the author created a receipts table instead of relying on a checkpoint file. The table uses the content of the work as the primary key, ensuring that each call is uniquely identified. The author also introduced the idem_key function to generate a unique identifier for each item. By marking items as claimed before the call to the model and updating the state accordingly, the author solved the problem of repeated work and latency during the restart.