A Green Suite Can Still Lie: Mutation-Test Your Fixtures Before You Trust an Agent Patch

An agent patch can pass all tests in a repository while still being incorrect. Fixtures are part of the "oracle" that determines the expected behavior, but they can be lazy or incomplete. If the expected value in a fixture never changes, the test suite can approve code that breaks the case the fixture overlooked. This is why evaluating agent-generated patches is crucial.

The process consists of three phases. Phase 1 involves running property checks before anything else. Property checks involve writing an invariant that must hold for all inputs, then feeding the function random inputs. This helps ensure the patch does not introduce or lose ranges, and that no overlapping intervals remain.

Phase 2 focuses on mutation testing. This technique involves taking each fixture, applying a small change that should alter the test outcome, and running the test suite. If the suite still succeeds, the fixture is deemed blind and ineffective at catching changes in behavior.

Phase 3 involves freezing flaky tests only after the first two phases pass. This ensures that any tests that intermittently fail are thoroughly investigated before being deemed a problem with the agent patch.

MonkeyCode offers a free server option for running this probe, and their free model access is useful for drafting the invariant skeleton from a patch's diff. It is important to remember that golden fixtures, which assert one output for one input, can be weak oracles. They catch regressions on paths you already know about, but fail to invent new cases. If the fixture never expected an edge case, the patched code will still return the expected output, leading to false confidence in the patch's correctness.