{
  "id": 5678174,
  "title": "Shadow-Compare the Agent Patch. Merge Only Classified Divergences.",
  "url": "https://urgent.news/2026/09/05/shadow-compare-the-agent-patch-merge-only-classified-divergences",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-05T00:40:37.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/datacpp_8185/shadow-compare-the-agent-patch-merge-only-classified-divergences-kgm"
  },
  "original_language": "en",
  "account": "A green test run does not represent a behavior specification. An agent patch may maintain all current assertions while modifying encodings, error types, empty input handling, or stdout bytes. Compare the candidate against a static baseline on the same dataset through shadow-comparison. Only merge after classifying every divergence into an accepted-delta ledger. This document outlines a testing workflow, not a model benchmark. The accompanying harness is labeled as a tentative, executable pattern. It does not guarantee production speeds, model names, or pass rates. The green CI may overlook agent patches, as they optimize for visible tests. Hidden behavior exists in unnamed branches, such as trailing newlines, NaN keys, timezone-naive timestamps, and None versus empty lists. These are easy to modify but difficult to detect post-merge. A dual-run gate uses the old artifact as the standard for unidentified behavior. Specified behavior should remain in regular tests. The ledger records both purposeful and rejected differences. Do not rely on this as a replacement for code review; instead, use it as a tool to minimize manual review. The contract consists of three files: baseline (a locked copy, wheel, or container hash), candidate (the agent patch applied to the same reference), and delta_ledger.yaml (previous classified output differences, identified by fixture ID). The ideal layout includes shadow/ and corpus/ directories for deterministic fixtures only.",
  "summary": "A green test run is not a behavior spec. An agent patch can keep every existing assertion passing and still change encodings, error types, empty-input handling, or the bytes written to stdout. Shadow-compare the candidate against a frozen baseline on the same corpus. Merge only after every divergence is classified in an accepted-delta ledger. This article is a testing workflow, not a model…",
  "key_points": [
    "Shadow-comparison validates agent patch against static baseline on same dataset.",
    "Only merge after classifying every divergence into accepted-delta ledger.",
    "Ledger records purposeful and rejected differences for minimized manual review."
  ],
  "editors_take": "This testing workflow shifts focus from passing automated checks to classifying and documenting behavior changes, treating policymakers as gatekeepers and code reviewers as key stakeholders in the merge process.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}