Your Retry Budget Is Not a Safety Net
*Second in a series on The Factory. Previously: The Factory That Merged 37 Tasks . The harness is at github.com/frozer/factory . The public description of my task harness ends on a claim: a packet that's wrong about the world fails identically on every retry. That sentence cost me four dead tasks and nine commits spent repairing task definitions instead of writing code. It reads like something…
A retry budget is often viewed as a safety net for reliability, but it can actually exacerbate bugs rather than solve them. The key is that a retry doesn't provide a new situation or environment for the model to work in. It simply attempts the same task again, potentially failing in the same way. This is because the underlying packet, data, and instructions remain unchanged.
The source material outlines a few examples of this phenomenon, including a national census dataset loader that failed on three attempts due to incorrect file paths and structures, and another task that suffered from a schema mismatch and design collision when loading multiple packets. The solution proposed is to implement a pre-screening check for packets before they are processed by the model.
This involves creating an assessor that checks the packet against a ground-truth specification, identifying any held, violated, or unverifiable claims. If a packet fails any of these checks, it should be refused and not processed further. This approach aims to catch defects early in the process, rather than allowing them to persist and be repeated through multiple retry attempts.
The author found this method effective, as it prevented several known-bad packets from being processed and identified a live bug in merged code that had gone unnoticed by tests or reviews.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
