What stops your agent shipping garbage?
The tests, in principle. But the agent wrote those too, out of the same understanding of the task, so they agree with the code even when the code is wrong. Why you care: this is the difference between an agent that checks its own work and one that confirms it, and the setup is a file, a runner and one CI job. The test is the same idea written twice: once as behaviour, once as an assertion about…
The tests conducted in this scenario reveal the difference between an agent that checks its own work and one that merely confirms it. The process involves creating a file, a runner, and a CI job with a single job to compare the outcomes. The key lies in having the agent intentionally break its own test and report what the test does.
By mutating the test code and observing the results, we can determine the effectiveness of the test. If a placebo is inserted in the test and the test still passes, it indicates a flawed test that cannot differentiate between correct and incorrect implementations. On the other hand, if a placebo causes the test to fail, it signifies a robust test capable of catching errors.
The article also highlights the importance of writing placebo proofs for the repository, adding a check in the CLAUDE.md file, creating a committed proofs file, and writing a runner to ensure each edit is an exact anchor plus replacement. Additionally, it emphasizes the need to identify and fix the weakest checks first, as they are the places where a check exists, passes, and would not notice if the guarded implementation broke.
By following these steps, the agent can verify its own work rather than merely confirming it, leading to more reliable and effective tests.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
