Five times our coding agents said "it works", and it didn't
I run a small open-source lab in Switzerland. Most of the code in our repositories is written by AI coding agents, from more than one vendor, working in parallel across about a dozen projects. They coordinate through a local hub we built for that purpose, and every change is supposed to be checked by a different agent before it lands. "Supposed to" is the interesting part. Over the past months…
Five instances were reported where coding agents alleged success despite failure. Each incident stemmed from differing measurement issues. In the first case, an agent claimed all tests passed on a subset, but the CI system later failed. The agent had only executed a limited set of tests, not the entire suite. The second incident occurred during mutation testing, where a file was restored from a backup, but the test still detected mutated behavior.
The mutation was size-neutral, causing the test to fail despite the file's integrity. The third incident involved a grep command reporting a message watcher, but it was not actually active. The agent's status was derived from a pattern match, not a definitive check, leading to incorrect assumptions. The fourth incident involved a green browser suite against an incorrect build.
The agent started a new server and stopped the old one, but the process listening on the port was not the intended one. This resulted in the test suite appearing green against an undesired build. Lastly, an agent landed the wrong commit, assuming a typo when copying a SHA. Despite having identical byte trees, the two commits differed in authorship information, impacting provenance.
A common denominator across these instances was the lack of evidence to support the agents' claims. The lessons learned emphasize the importance of verifying claims with different agents, running exact commands, and maintaining an event log for accountability. These tools can improve the visibility of mistakes and enhance the reliability of parallel agent operations.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
