My AWS AI Agent Passed Its Tests. It Still Missed a Check.
I gave my AI agent a small deployment history and asked how often the checks were unhealthy. It wrote the Python, ran its own tests, and came back with the right answer. Then I read the tests. They passed, but they didn't cover everything I'd asked for. Some context: this was a small experiment in my personal AWS lab, with made-up deployment data. There's no production incident behind it. A…
The AI agent was given a small deployment history and asked how often checks were unhealthy. It successfully wrote Python code, ran its own tests, and returned the correct answer. However, the tests did not cover all the aspects the reporter had asked for. The deployment history consisted of twelve checks taken five minutes apart, with each row recording the status, expected and running application versions, instance counts, and active alarms.
The agent calculated that five checks were unhealthy, representing 41.7% of the total samples. It also determined that there were two completed unhealthy periods, lasting 15 minutes and 10 minutes respectively. The agent generated tests to verify the period counts and durations, but they did not explicitly check every expected start and end timestamp.
Despite the tests passing, the reporter realized that they only proved the assertions written by the agent, not every requirement. The experiment highlighted the importance of reading the generated tests to understand what was actually proven by the agent's analysis.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
