I expected AI agents to write hollow tests. They didn't.
A test that cannot fail looks exactly like a test that works. It is green in CI, it counts toward coverage, and it protects nothing. I wrote hollowtest to find them. It reads your test files and flags tests that would keep passing if the code under test were broken: npx hollowtest What a hollow test looks like Some are obvious once you see them: it ( ' applies the coupon ' , () => { applyCoupon (…
The article discusses how hollow test cases can be difficult to spot, even in well-maintained codebases. It explains that hollow tests are tests that will always pass, regardless of any changes made to the underlying code. The author wrote a tool called hollowtest to identify these types of tests across various programming languages and testing frameworks.
The tool works by analyzing test code and flagging tests that would still pass even if the code being tested were broken. The article notes that hollow tests often involve unnecessary assertions, missing parentheses, or improper handling of asynchronous code. It also mentions a study where an AI coding agent, Claude Code, was tested to see if it would write hollow tests.
The study found that the agent was able to identify 31 out of 32 bugs planted in small JavaScript modules when writing tests. However, the agent refused to weaken tests when explicitly pressured to do so, and a plugin designed to catch these types of tests did not provide any measurable benefit in this scenario. The author concludes by stating that hollow tests are a human habit first, and notes that the findings of hollow tests in the wild mostly predate coding agents.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
