Diff the Behavior, Not the Code: A Differential Gate for Agent Patches

An article titled "Diff the Behavior, Not the Code: A Differential Gate for Agent Patches" explores the importance of evaluating software patches based on their behavior rather than just their code changes. Traditional unit tests often fail to capture the effects of agent patches that introduce caches, retries, ordering, and shared state. To address this limitation, the article introduces a differential testing approach that compares the outputs of old and new implementations when given the same inputs.

The core idea behind the differential gate is to run the old code and the new code on identical inputs, then compare their outputs. If the outputs match, it suggests that the patch does not alter the expected behavior of the system. The article presents two equivalence properties: call-level equivalence and sequence-level equivalence.

Call-level equivalence ensures that individual calls to the patched and unpatched implementations produce the same results. Sequence-level equivalence goes a step further by verifying that the sequence of outputs generated by the patched implementation matches the sequence produced by the unpatched implementation.

To implement the differential gate, the article suggests a step-by-step process. First, the baseline code is frozen by building the current code and recording its outputs for every input fixture. These outputs are hashed and stored as the truth that the gate aims to protect. The hashed values should be kept separate from the agent's write path to maintain the integrity of the truth.

Next, a hostile corpus is generated to include edge cases that agents often exploit. This includes inputs with empty content, maximum length, punctuation, and repeated keys. The corpus is generated using a Python script that creates 2000 random inputs with varying lengths and complexities. The repeated-key block is crucial as it forces the testing of cache or state bugs, which unit tests may not detect.

Finally, the differential gate is executed by replaying the frozen corpus against the candidate binary and comparing the outputs byte-for-byte with the frozen baseline. No additional assertions beyond equality are made, ensuring a fair evaluation of the patch's behavior. If any discrepancy is found, the gate fails, indicating that the patch has introduced unintended changes.

By focusing on the behavior of software patches rather than just their code changes, the differential gate provides a more reliable method for ensuring that agent patches do not unintentionally alter the expected behavior of a system. This approach helps maintain the integrity of the software and prevents potential issues arising from hidden state or other side effects introduced by agent-generated tests.