One Command to Verify an Agent Patch: Seeded Hypothesis, Fixture Contracts, and a TTL Quarantine

A single command should verify an agent-patched Python project. The verification should be a repeatable experiment, not just a mental checklist. This post describes a three-part process to accomplish this: seeded property checks, fixture contracts, and a TTL quarantine for flaky tests.

First, seeded property checks generate test cases using the Hypothesis library. A fixed set of examples is insufficient to guarantee the code's correctness, especially when dealing with boundary conditions. The provided example demonstrates a property test for a discount helper function that checks invariants and ensures the result is non-negative and limited to two decimal places. The seed value in the test ensures reproducibility of the test runs.

Second, fixture contracts define a contract between the test and the domain. When an agent patch alters the fixture loader, the fixture contract catches the issue before any domain assertions run. The provided example shows how to create a contract for a ledger entry fixture, specifying required fields and semantic checks. By incorporating this contract into the test suite, regressions caused by changes in the fixture loader are detected early.

Lastly, a TTL quarantine is employed to handle flaky tests. Flaky tests can be costly and distracting when left in the test suite. The quarantine allows a flaky test to remain inactive for a fixed duration, after which it is presumed fixed and re-enabled. The provided example shows how to create a quarantine file that stores the excluded tests and their expiration dates. The test runner reads this quarantine file and skips the tests that have expired their quarantine period.

All three rungs - seeded property checks, fixture contracts, and TTL quarantine - work together in a single command using pytest and hypothesis libraries. These tests produce standard pytest output and do not trust generated tests merely because they were created by a model. The only dependencies required are pytest and hypothesis, and the entire process is deterministic, ensuring reliable and maintainable testing.