{
  "id": 4994978,
  "title": "Give Every Agent Patch a Regression Budget, Not Just a Green Check",
  "url": "https://urgent.news/2026/09/02/give-every-agent-patch-a-regression-budget-not-just-a-green-check",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-02T02:37:55.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/datacpp_8185/give-every-agent-patch-a-regression-budget-not-just-a-green-check-9pa"
  },
  "original_language": "en",
  "account": "A green check in CI is not sufficient to confirm the quality of an agent-generated patch. This article explains how to establish a regression budget for these patches, which includes a fixed number of properties, fixtures, and flaky-test quarantine slots that the patch must pass before merging. The author demonstrates how to define this budget using a small YAML file and a shell script that runs three verification phases: properties, fixtures, and quarantine. The budget model consists of three buckets with cost limits measured in compute minutes. Properties have 200 cases per property with no counterexample allowed, fixtures have 20 contracts per patch with all checks required to pass, and quarantine has 5 entries with no failed test allowed twice. Implementing the budget gate involves a shell script that reads the budget YAML file and stops the verification process if any phase exceeds its cap. A fixture inventory artifact is provided to track which fixtures are impacted by the patch, allowing only changed fixtures to be verified in the fixtures bucket. The author emphasizes that this budgeting idea works with any CI, provided it has free model access and a free server tier. They run the gate on a free server instance and report a total of 20 minutes per patch, which is long enough for human supervision but short enough to be practical. However, the budget model assumes deterministic tests and a small flaky test rate. If tests are flaky or the suite's complexity requires more advanced infrastructure, the author suggests using a purpose-built platform and keeping the budget idea. The budget is not intended for security-critical modules, where exhaustive verification is necessary. The article concludes with a recommendation to test the budget against a deliberately broken patch once per month to ensure it is still effective.",
  "summary": "A green CI run is not a verdict. It is a photograph taken at one moment, under one seed, with one set of fixtures. When the photographer is an agent that rewrites five files in a single pass, you need more than a photograph — you need a budget. This article shows how to define a regression budget for agent-generated patches: a fixed number of properties, fixture contracts, and flaky-test…",
  "key_points": [],
  "editors_take": "Establishing a regression budget for agent-generated patches shifts quality assurance from a simple pass-fail check to a more nuanced evaluation that balances thoroughness with practical resource constraints.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}