Opinion: Repo Tests Are a Compromised Oracle for AI Patches — Probe the Contract the Model Never Saw
A green CI badge measures one thing only: the tests you already wrote still pass. It does not measure whether an AI-generated patch preserved the behaviors you never wrote down. When the model read those tests as part of its prompt context, the badge is scoring a target the patch was already aiming at. My position is direct: for free-model patches, the repo test suite is a compromised oracle, and…
The article argues that CI badge tests are not reliable indicators of AI-generated patch behavior. These tests only measure if the tests you wrote still pass, not whether the patch preserves the behaviors you didn't document. The model sees the tests as part of its prompt context, so a patch that passes the tests only means it matches patterns, not that it truly satisfies the behavior.
The article asserts that the test suite is compromised and should not be used to judge the patch. Instead, a standalone contract probe that runs outside the repository and contains the invariants your team debugged at 3 a.m. should be used. This probe exercises a running instance and compares the baseline and patched instances, emitting a JSON report.
A failed probe on the baseline indicates a pre-existing issue, while a failed probe on the patched instance signals a patch regression. The diff between the baseline and patched instances is the only diff that matters.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
