{
  "id": 4955052,
  "title": "Four Ways a Batch Runner Can Believe It Already Finished",
  "url": "https://urgent.news/2026/09/01/four-ways-a-batch-runner-can-believe-it-already-finished",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-01T22:11:49.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/jula-markova/four-ways-a-batch-runner-can-believe-it-already-finished-4kcd"
  },
  "original_language": "en",
  "account": "On 16th March 2026, the batch runner for our content pipeline reached a single commit. This commit included a resume flag, a state log, error classification, a quota pause, and a lock file. The feature was present from the beginning, yet it took three months to work properly. This delay is the focus of the article. Resumable AI systems typically refer to agents remembering conversations across turns, threads, or replaying a graph from a saved node. However, this is not the only aspect. A batch pipeline must know which work is already completed, across separate runs and different machines. The process of building this functionality was not as straightforward as it seems. Eleven days before the runner existed, we had implemented a circuit breaker (stop after three consecutive quota failures) and a shutdown path that terminates child processes when the parent is interrupted, preventing orphaning. These were survival mechanisms, not true resumability. Four days after the runner, we introduced a retry mechanism that continued a partially-written file instead of regenerating it, a common misunderstanding between retry patterns and true resumability. The runner tracks two distinct axes: the entity state and the failure category. The entity can be in one of five states: pending (not started or reset after a dead run), running (claimed by the current run), done (all planned output exists), failed (stopped for a reason that waiting won't fix), or paused (stopped for a reason that waiting will fix). Additionally, failures can be classified into six categories: timeout, quota, validation, phase_error, planning_error, or unknown. These categories help differentiate between different types of failures. A failure with a quota category is not a defect but a system telling the batch to come back later. Validation errors are different - they are deterministic and stop the run for human intervention. A notable detail only appears after a run has been killed: the entities are frozen in the running state, and nothing moves them forward on their own. To reclaim these entities, a separate process is required. This process must cover entity state, not the lock file left behind by the dead run. We discovered this the hard way. In June, we identified four separate issues within the same mechanism, all occurring in one day. Each issue required a unique fix, demonstrating that resumability is not an inherent feature but a deliberate design choice.",
  "summary": "On 16 March 2026 the batch runner for our content pipeline landed in a single commit. That commit already contained a resume flag, a state log with per-item status, error classification, a quota pause, and a lock file. Resumability was not bolted on after it hurt. It was there on day one, and it still did not work for another three months. That gap is the article. When people write about…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}