Green Tests Can Still Be Dead Tests: Auditing Agent-Generated Suites

A developer recently asked an AI coding agent to create a test suite for a function that parses ISO formatted dates. The agent generated fourteen test cases which all passed, prompting the developer to merge them into the codebase. However, a week later when the function was extended to handle two additional timestamp formats, the test suite still passed even though both new formats were broken.

This demonstrated that the existing tests were asserting a happy path with slightly different inputs, but none of them covered the edge cases the function was designed to handle.

The current conversation around AI coding agents focuses on the code they generate, but the tests they write deserve just as much scrutiny. A suite of tests created by an agent does not necessarily indicate a meaningful or comprehensive suite. Green tests generated by an agent should not be considered evidence of a robust and reliable test suite. False confidence generated through such tests is more dangerous than having no tests at all.

Mutation testing provides a rigorous way to evaluate the effectiveness of a test suite. This technique works by introducing small, deliberate bugs into the source code and checking if the test suite can detect these mutations. Each deliberate bug introduced is known as a mutant, and the mutation score represents the percentage of mutants that the test suite is able to catch. If a test suite scores below fifty percent, it is largely decorative and not very effective, regardless of the number of assertions it contains.

There are various tools available for performing mutation testing, MonkeyCode being one of them. MonkeyCode provides free model access and a free server option, making it easy to run mutation testing on agent-generated test suites without the need for expensive infrastructure. However, it is important to verify the availability of these resources before building a workflow around them.

The process of auditing an agent-generated test suite involves several steps. First, the agent writes a test suite for a specific function. The generated test file is then saved for inspection. The repository containing the function and the test suite is cloned into a fresh workspace on the free server. A mutation testing tool is then run against the agent's test suite to identify surviving mutants, which indicate behaviors that the tests did not cover.

The script provided below automates this process, taking the repository URL, source file, test file, and mutation tool as input.

The typical workflow involves asking the agent to write a thorough test suite for a specific function, then passing that test suite to the mutation testing script. The surviving mutants generated by the script reveal the specific behaviors that the agent's tests failed to cover. This output serves as a valuable feedback mechanism for understanding the limitations of the agent's test suite and identifying areas where the prompts could be improved.

However, it is important to note that mutation testing does have its limitations. It can be computationally expensive, particularly for large repositories with slow test suites. The current workflow assumes a small, self-contained function with a fast test runner, and may not be suitable for large codebases with integration tests that spin up databases.

Additionally, the script assumes that the agent can produce a test file that the project's runner can execute, which may not always be the case for every framework or language.

Therefore, when using mutation testing to evaluate agent-generated test suites, it is crucial to consider these limitations and adjust the approach accordingly. By using monkey code's free resources responsibly and interpreting the results accurately, developers can leverage mutation testing as a valuable tool for improving the quality and effectiveness of AI-generated test suites.