Blame the Model Last: An Attribution FAQ
An illustrative scenario, labeled as such Imagine a repository where two integration tests fail, so a developer hands an agent a single sentence: make the suite pass. The agent returns a diff touching three files, the tests still fail, and the postmortem line writes itself: the free model was too weak. That conclusion may be correct, but nothing in the run so far actually supports it, because the…
To determine the root cause of a failing test suite, it is essential to record the inputs before judging the output. Simply looking at a patch alone is not enough, as it can originate from any earlier step in the pipeline. The mental model should be divided into five distinct layers: task specification, context assembly, execution environment, model, and verification. Each layer can fail independently and must be examined separately.
Wall-clock time measurements alone do not provide a clear indication of where the bottleneck lies. They encompass queueing, cold start, dependency installation, model latency, and test runtime. By capturing per-phase timings, one can identify which phase is responsible for the delay. Running the same command in a different environment serves as a control to understand if the issue lies within the repository or the environment.
This hypothesis-generating approach is preferable to benchmarking, as a single pair of runs is insufficient to support a performance claim.
A clean diff does not guarantee that the work is complete, as syntactically valid diffs can be semantically empty. An agent may edit generated artifacts that are later overwriting, add defensive branches that never execute, or weaken assertions without appearing as merge conflicts. The verification surface should be treated as part of the reviewed surface, and any change to the test command, CI configuration, or assertions deserves the same scrutiny as the fix.
A heuristic can be used to identify patterns, but it may produce false positives even on legitimate test work.
Swapping models should not be the first step, as it appears cheap but has an invisible cost of re-reading fresh diff against unfamiliar context. Replaying the same task with complete logs and a pinned toolchain is usually less attention-consuming and yields a sharper answer. However, if the context and environment were established to be identical, a model comparison becomes genuinely informative.
On platforms with limited resources, such as free tiers, the scarce resource is not the generation itself, but the ability to reconstruct what the run actually saw. Using a disciplined replay preserves this ability.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
