Properties Without Witnesses Are Decorations: A Falsifiability Gate for Agent Patches
A property that passes on the patched tree and on the pre-patch tree carries no information. That is the first thing to measure, before you trust any property an agent wrote for you. Most agent-written property tests fail this way. The agent reads the diff, writes an invariant that the new code obviously satisfies, and the suite goes green in both trees. The test is not wrong. It is simply unable…
The article discusses the importance of having a gate with specific rules to evaluate the effectiveness of properties written by agents. The first rule requires a committed witness for each property, which is a concrete input that proves the property can fail. This witness must be kept in the repository, reviewable, and cannot be invented by the agent during runtime.
The second rule measures the detection power of the property on the baseline, not on the patched tree. This helps determine if the property generalizes beyond the specific input used in the witness. The third rule isolates fixtures per sample and verifies them using transcript hashes to ensure that the setup is deterministic and does not leak state between samples.
These rules aim to prevent properties that simply pass without providing any meaningful information or detectable differences between pre-patch and patched trees, effectively making them mere decorations rather than useful checks.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
