I Shipped My Code Gate to a Stranger's Machine. Its First Bug Report Fixed the Evaluator Itself.

I built a structural gate that prevents AI coding agents from writing files before they are approved. This gate enforces strict limits on file size, imports, and human overrides. The gate's first real-world test occurred when it ran on someone else's machine, processing a TypeScript project and attached evaluation questions.

During testing, the gate issued three correct refusals, one false positive, and a list of corrections. Each correction was implemented within a day. The same machine then verified all the corrections against the code that produced them, demonstrating a closed feedback loop.

Surprisingly, the gate generated a false positive when it refused to create a file in a session scratchpad due to a rule about directory size. The gate had no concept of a non-project directory, so it applied a project-level rule to a non-project directory, resulting in a correct but unintended refusal. The fix for this false positive involved adding a check to ensure the directory is under a package.json or .git before applying architectural rules.

One critical oversight during the evaluation was that the gate was inactive for two days, and no one noticed it wasn't working. This oversight could have prevented valuable feedback from being collected. The report identified this as a weak point in the tool's distribution story.

The most significant finding was that the gate was counting type-only imports in the 7-import budget. TypeScript can remove type-only imports at compile time, so these imports should not count against the budget. This oversight caused a few refusals to occur at the 8-versus-7 import threshold, distorting the tool's decision-making process.

To address this issue, the gate was updated to exclude type-only imports from the import budget count. The changes were verified by running the evaluator against the same codebase and the same scratchpad that originally produced the findings. The evaluation was repeated, confirming that the issue had been resolved.

In summary, this experience demonstrates the importance of thorough testing, robust error handling, and continuous verification in developing reliable software tools. The gate's ability to identify and fix issues quickly and effectively showcases its potential as a valuable tool for managing AI coding agents.