Properties First, Fixtures Second, Flakes Never: A Test Contract for Agent Patches
An agent patch is a hypothesis. A test suite is the only evidence a reviewer gets. Most suites fail at that job in three reproducible ways: assertions the agent can reverse-engineer, fixtures that regenerate and drift, and flaky tests that flap between green and red without a single commit. This article is a three-layer contract that closes all three: property checks for invariants, hash-pinned…
The article presents a three-layer contract to address the common issues of assertions, fixtures, and flaky tests in test suites. Layer 1 focuses on property checks that transform values into relationships, ensuring that outputs do not violate the module's documented contract for any input. This approach prevents agents from merely memorizing answers and requires them to change behavior when violations occur.
Layer 2 tackles fixture drift by generating fixtures at test time, hashing them, and asserting before the suite runs. This technique ensures determinism and visibility of differences, preventing hidden regression issues. Layer 3 introduces a flake quarantine that runs tests before the agent sees the suite, avoiding flaky test failures.
By combining these three layers, the contract aims to establish a reliable and verifiable testing process, improving the overall quality of software development.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
