{
  "id": 19565581,
  "title": "Your agent's regression test should fail on the old code",
  "url": "https://urgent.news/2026/11/11/your-agents-regression-test-should-fail-on-the-old-code",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-11T23:00:39.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/hefty_69a4c2d631c9dd70724/your-agents-regression-test-should-fail-on-the-old-code-5gcm"
  },
  "original_language": "en",
  "account": "An agent's regression test should not pass on outdated code, as it provides a misleading answer to whether a fix successfully resolved a bug. To ensure compatibility, compare the application's behavior against both the original code and the proposed fix using the same checks. The old code should fail in the manner intended for repair, while the proposed solution should pass without altering the acceptance criteria. Clearly define the target behavior before evaluating the fix. For example, if a field incorrectly accepts blank input as zero, the requirement is to reject blank input and still accept valid decimal amounts. This distinction is crucial; a \"reject everything\" approach would satisfy a poorly written blank-input test but break the feature. Before accepting any modifications, carefully review expectations, the boundary of rejection, and the test's focus on a specific interface exposed by both revisions. The regression test should remain constant across reviewed revisions, and its content and fixtures should be held constant during evaluation. Run the regression test against each revision's application code, keeping the environment constant. A failing assertion when blank input is present confirms the expected outcome, while a passing assertion when valid input remains accepted further validates the fix. Examine the base failure to ensure it is not due to issues like missing imports, broken fixtures, or runner crashes. The test should reach the relevant assertion and fail appropriately. If both revisions pass the blank-input test, it does not demonstrate the regression being fixed. Inspect the fixture, execution path, and assertion before drawing conclusions. This rule focuses on testing a specific scenario, and passing the test does not guarantee comprehensive coverage. Test changes should undergo their own review process. Look for removed assertions, newly skipped cases, fixtures that avoid the problematic input, or suppressed type checks. Changes that simply stop selecting the relevant test are not inherently suspicious but should be reviewed when requirements change. The \"trustbut\" workflow recommends rerunning tests in a clean environment, comparing changed tests against pre-change code, and flagging patterns such as skipped tests and removed assertions. This approach aids verification but does not guarantee the test's completeness. Always verify the claim separately and check the proposed solution against it. Trustbut's structured receipt system can make verification more manageable, but it is not a guarantee of every possible weak point. Run the test in a controlled environment using a clean checkout, but recognize that dependencies, test commands, and credentials may still pose risks. A fresh worktree identifies the source state, but it does not ensure a hermetic environment or protect against untrusted code or personal credentials. Use a disposable runner with only the necessary access for the check. Record the runtime and dependency state, preserving any limitations that prevent the base from installing dependencies or requiring compatibility adjustments. Silently altering the setup until something fails makes the comparison more challenging to interpret. Request a receipt containing the base and candidate revision IDs, exact regression-test and fixture content or blob IDs, command, selected tests, runtime and dependency state, intended behavioral failure on the base, the actual failure location on the proposed solution, results from valid-input controls, skips, and setup failures. Exclude unrun checks and infrastructure failures from the pass column. Before merging, ensure the original defect that the reviewed check caught is still present in the broader candidate suite. A green badge is more trustworthy when you know what would have triggered a red result.",
  "summary": "“The tests pass” is an incomplete answer to “did this fix the bug?” An agent can change the implementation and the tests together, then hand you a green suite that never exercises the original failure. For a compatible bug-fix regression test, ask for a comparison: run the same check against the old code and the proposed fix. The old code should fail for the behavior you meant to repair. The…",
  "key_points": [
    "Agent's regression test should fail on outdated code",
    "Compare application behavior against original code and proposed fix",
    "Define target behavior before evaluating the fix"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}