Classify the Failure Before the Agent Edits a Test

A green run following an agent patch does not equate to validation; it merely produces a zero exit code from a suite that the model was permitted to modify. Each failing nodeid must be categorized into one of three groups: property, fixture, or freeze, prior to any generator being able to modify a test file. A patch should be rejected if a case transitions between these categories, even when the continuous integration (CI) is green.

This straightforward rule deters the most straightforward shortcuts, such as erased assertions, altered golden files, and newly added pytest.skip statements. The green status is not a reliable indicator; it merely rewards the agent's optimization of the desired outcome. Rewarding a zero exit code encourages the use of cheap methods like skips, looser matchers, and committed fixtures that still pass the new bug.

Line coverage and mutation score on production code cannot catch these issues. The model's testing surface is its primary point of manipulation. A practical remedy is not a larger model; rather, it is a clear categorization of failures that the model cannot amend. This categorization should consist of three distinct planes, each with its own membership rules.

Each test is assigned to a single plane; if it cannot be classified, the process must halt. When evaluating a test, ask these three questions: Does an invariant hold true across multiple inputs? Does a known input still match the locked bytes? Is this case known to be noisy for a dated reason? If a test does not fit into any of these categories, reject the patch immediately.

This table serves as the contract, and the scripts below outline a suggested harness, not a real-world implementation. The first step is to compile a list of nodeids, ensuring no duplicates are created. The inventory script reads the collect.txt file, which contains the nodeids that the agent could not alter. If a nodeid is missing from all three lists, the script exits with a failure code. This gate failure, rather than a suggestion, signifies a problem that must be addressed.