SpecGuard: When AI Coding Agents Write Tests That Approve Bugs
Ask a coding agent to write checks for a requirement and it will often copy what the existing code does instead. In SpecGuardBench, the best agent wrote a faithful validation harness for only 63% of tasks. When the reference code was buggy, the agents' faithfulness fell by 19 to 31 points, because they wrote checks that approved the bug. Here is a free, in-browser way to test any model for this.…
A coding agent tasked with writing tests based on a requirement often ends up copying the existing code instead of focusing on the given specification. In the SpecGuardBench, a benchmark that evaluates coding agents' ability to construct consistent validation harnesses from natural language specifications, only 63% of tasks yielded a faithful harness from the best agent.
When the reference code contained a bug, the agents' faithfulness to the requirement declined by 19 to 31 points, as they produced checks that approved the bug.
To test a model's susceptibility to this issue, researchers provided a one-line requirement—slugify(title) must return a string containing only lowercase letters, numbers, hyphens, and no leading or trailing hyphens. Using this requirement, the coding agent was given the current implementation, the existing test, and asked to create a validation harness.
The straightforward approach would be to run the current code, record its outputs, and assert that any other implementation produces the same result. While this approach may seem thorough, it has two significant flaws: it inadvertently approves buggy code and rejects valid alternatives.
This issue becomes more pressing as coding agents increasingly handle code creation. A benchmark study, SpecGuardBench (arXiv:2610.16870), tested 291 Python tasks covering eight types of verification logic with correct, buggy, and valid alternative implementations. The benchmark found that coding agents had a faithfulness score ranging from 45% to 81% when working from the requirement, but fell to 21% to 46% when provided with a buggy reference implementation. This demonstrates that agents trust the reference code more than the given requirement.
To address this problem, the authors developed SpecGuard, a benchmark designed for browser use. SpecGuard presents users with a requirement, a way to call the code, a reference implementation, and an existing test. Hidden from the user are a correct implementation, a buggy one that passes the existing test, and a valid alternative.
Users can then choose whether a local model or one of three baseline models writes the harnesses. The local model runs on the user's GPU without sending any data to a server, ensuring no harmful code is uploaded. Three additional baselines demonstrate how the scoring system works without requiring a GPU.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.