Six patch releases in four days: how my merge gate kept failing open, and what fixed it
Earlier this month I wrote about AdversaryGate , a merge gate for patches written by coding agents. It runs your tests on the baseline and on the patch and answers MERGE (exit 0), BLOCK (1) or INCONCLUSIVE (2). The rule I built it around: a measurement nobody produced can't clear a floor. Since then I shipped 3.0.0 through 3.0.6. Most of those releases exist because I kept attacking my own gate…
In the span of four days, six patch releases were made to address issues with the AdversaryGate merge gate for patches. The gate determines whether a patch should be merged based on test results, with possible outcomes being MERGE, BLOCK, or INCONCLUSIVE. The author discovered that the gate was failing open, meaning it was allowing patches that should have been blocked to pass through unchecked.
The problems stemmed from the gate's inability to measure certain inputs, causing it to silently skip them and result in a consistent MERGE outcome. Some examples include tests that read the code instead of running it, tests that hash the changed file and fail on any edit, and tests that compare the AST or bytecode but pass a comment canary. Other issues included binary files being marked as code, documentation being counted as code, and branches on the same line as their guards.
The author addressed these issues by implementing various fixes in the patch releases. These fixes included modifying how the gate measures inputs, ensuring that only actual code is considered, and adding documentation to clarify the gate's behavior. For instance, a test that reads the code was fixed in 3.0.0, a test that hashes the changed file was made to fail in 3.0.1, and a branch on the same line as its guard was addressed in 3.0.4.
Additionally, the author implemented a security measure to ensure that only changes to the product module, not imported through dynamic imports, are considered. This was done in 3.0.6. However, it is important to note that the gate still has limitations, such as only measuring Python and pytest, and not detecting changes made through dynamic imports.
The author has provided a comprehensive ledger detailing each bug found and how it was fixed, which can be found in the SECURITY.md file. To report any new issues or concerns, users are encouraged to do so privately as outlined in the document. The latest stable release of AdversaryGate is version 3.0.6.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
