Order Is a Hidden Input: Permutation-Sweeping an Agent-Patched Test Suite
An agent patch can pass your entire suite in the default order and still be order-coupled. The green run proves the suite works in one sequence, not that the tests are independent. Before I trust a machine-written patch, I want to know which tests only pass because something ran before them. That is a different question from "did the patch's tests touch the changed lines" or "are the assertions…
Order is a hidden input when assessing agent-patched test suites. A patch might appear stable in the default order, but it can still be order-coupled. This matters more than whether the patch affected the changed code or if the assertions are strong enough. The suite should be viewed as a stateful program, as agent patches are adept at altering that state.
Common culprits include modules with caches, fixtures that modify shared data, imports with side effects, and monkeypatches not properly undone. The following process outlines how to uncover these issues and prioritize fixes.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.