Codex or Claude wrote my tests — how do I actually trust them?
Short answer: don't trust them because they're green. AI writes tests that assert the code's current behavior — including the bugs. Earn the confidence by mutation testing: change the code on purpose and check the tests fail. The ones that stay green are lying to you. Last month I let Claude generate a test file for a pricing function. Forty tests, all passing, 94% coverage, took ninety seconds.…
The short answer is to not trust AI-generated tests, as they tend to pass even when the code contains bugs. To gain confidence in AI-generated tests, use mutation testing. This involves deliberately introducing bugs (mutants) into the code and running the tests against each mutated version. If a test suite fails to catch a mutated bug, it indicates a lack of proper testing.
Tools like mutmut for Python, Stryker for JavaScript/TypeScript, PITest for Java, and Stryker.NET for C# can automate this process. In a case study, AI-generated tests for a pricing function achieved 94% coverage but only caught 50% of injected bugs when subjected to mutation testing. This highlights the importance of treating surviving mutants as missing assertions and fixing them.
While AI can swiftly generate tests, mutation testing should be applied selectively to critical modules with high risk, such as payments or authentication. Remember, mutation testing verifies the ability of tests to detect code changes, not the correctness of the original specifications, and it does not address non-deterministic AI outputs or the quality of the original problem definition.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
