48-Hour Field Notes: The Batch Worker Vanished and My Re-Run Redid Every Item
Two days ago I started a batch of prompt evaluations, closed the laptop lid, and assumed the free server would keep chewing through the queue while I slept. It did not. When I opened the terminal the next morning, the worker was gone and the only output was a JSONL file that stopped in the middle of a record. Why do we keep trusting long-running processes we never instrumented? This is a field…
Two days ago, the author initiated a batch of prompt evaluations and closed their laptop, expecting the free server to continue processing the queue while they slept. Upon opening the terminal the following morning, they discovered the worker had vanished, leaving behind a partially completed JSONL file. This incident marks the beginning of a 48-hour investigation into the issues they encountered and the measures taken to rectify them.
The initial batch script, naive_batch.py, consisted of around thirty lines of code. It accessed MonkeyCode's free model, ran the script on the free server, and appended one JSON line for each result. At first glance, the loop appeared error-free. However, upon closer inspection, several underlying assumptions were revealed. The index i served as the sole identity for each item, the open file handle represented progress, and the loop had no awareness of completed tasks. If the machine were to fail, all the process knew would vanish along with it.
The breakdown of the script occurred in a specific order. The enumerate counter acted as the resume state, causing a fresh process to always start at zero and resend all data. Retry attempts lacked unique identifiers, leading to duplicate rows when timeout errors occurred during individual calls. Furthermore, writes were not atomic, and a worker terminated mid-write left an unparsable line in the file, causing the summary script to crash on the last record instead of skipping it.
Ultimately, none of these failures stemmed from model problems; they were all due to inadequate bookkeeping.
To address these issues, the author introduced a new version of the script, ledger_batch.py. This version assigned each work item a key derived from the normalized prompt, prompt version, and a model alias controlled by the user. Before sending any data, the script read the ledger file to ensure it was working with the most up-to-date information.
The ledger module, named ledger_batch.py, encapsulated the key generation and ledger management functionalities. By resuming based on content hash rather than hope, the script minimized the risk of duplicate or incomplete entries, ensuring a more reliable batch processing experience.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
