An Agent Patch Is a Hypothesis. Test It Like One.

An agent patch is essentially a hypothesis about the codebase. It makes three claims: the invariants still hold, the output changes are intentional, and the new failures are not random occurrences. Continuous Integration (CI) should validate all three claims before a human reviewer examines the change. The author emphasizes a three-checkpoint contract as the gate for patch acceptance: property delta, fixture delta, and flake freeze.

Each checkpoint addresses a question that a reviewer would naturally ask; the objective is to make these answers machine-checkable, allowing reviewers to allocate their time more effectively.

Generating a patch is becoming increasingly cost-effective, reducing the constraint on patch attempts. However, this does not reduce review time. If the cost of reviewing one patch exceeds generating ten patches, the agent will eventually prevail through persistence. This article, prepared as part of MonkeyCode's product outreach, utilizes model-agnostic principles.

The focus is on treating the free tier as an availability claim rather than a benchmark, ensuring the contract does not depend on specific model names, quotas, or uptime figures.

Checkpoint 1, property delta, involves running property tests on the merge base and the patch, then comparing the results. If any property that passed on the baseline fails on the patch, the merge is blocked. These property tests define invariants, such as an LRU cache never exceeding its capacity or a serializer returning the original data after serialization.

The property suite runs on both the baseline and the patch, comparing the outcomes using a short parser that processes JUnit XML reports. Any discrepancy triggers a failure, indicating a change in behavior that violates the standing contract with the rest of the codebase.

Checkpoint 2, fixture delta, focuses on fixtures as fixed inputs with locked outputs. These fixtures help identify deliberate shifts in format, ordering, or rounding that still preserve the invariants. Any changed fixture must be named in the FIXTURES.md file, submitted in the same commit that updates it, accompanied by a brief justification.

The checkpoint reviews the changed files and checks the manifest to ensure all modified fixtures are accounted for. Agents that modify golden files without proper documentation do not truly address the issue; they merely alter the testing criteria. The manifest ensures that the intent of these changes is transparent and visible to reviewers during the review process.

Checkpoint 3, flake freeze, addresses flaky tests that produce inconsistent results. A flaky test can contaminate the results of property and fixture tests, making subsequent evaluations unreliable. The freeze mechanism involves quarantining any test that fails during a gate run. If a test fails even once during the gate, it remains in quarantine, preventing the merge.

Only after three consecutive clean runs, with no failures in the quarantine, can the merge proceed. This harsh rule ensures that only stable, reliable tests are allowed to progress, maintaining the integrity of the first two checkpoints.

The full gate consists of these three checkpoints: property delta, fixture delta, and flake freeze. Each checkpoint answers a critical question that a reviewer would naturally ask. By making these answers machine-checkable, the review process becomes more efficient and objective. The article concludes by emphasizing the model-agnostic nature of the contract, which remains valid regardless of the underlying model, quota, or uptime conditions.

This approach aims to streamline the review process, allocate reviewer time more effectively, and maintain the quality and stability of the codebase through automated, repeatable checks.