{
  "id": 17134,
  "title": "Your New Eval Rule Is Untested Code Guarding Production",
  "url": "https://urgent.news/2026/08/02/your-new-eval-rule-is-untested-code-guarding-production",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-02T01:01:44.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/saurav_bhattacharya/your-new-eval-rule-is-untested-code-guarding-production-2a1p"
  },
  "original_language": "en",
  "account": "A newly created *eval* rule successfully detected a production failure, but implementing it as untested code presents a significant risk. Treat*ing evals like ordinary programming* is misleading, as the true challenge lies in ensuring the rule accurately differentiates between acceptable and unacceptable outputs.\n\nThe most insidious issue arises from a false-negative gate: a rule with a regex bug that matches nothing, passing all tests while remaining ineffective. For three weeks, the dashboard remains green, providing a false sense of security. Only when a customer encounters the exact failure that the rule was meant to prevent does the true failure become apparent, with the rule never activating once. Consequently, a green eval does not guarantee a healthy agent; it merely indicates either a healthy agent or a flawed eval, with no means to differentiate without pre-fed traces known to produce errors.\n\nEvaluations, though not pure code, are still susceptible to the same testing principles. As they govern production, they should undergo rigorous testing akin to other code. This involves maintaining a corpus of labeled traces and asserting that each eval produces the expected verdict. Newly added rules must fail on known-bad traces and pass on known-good ones before being considered valid.\n\nRanking eval evidence should prioritize *independence*—the likelihood of the signal being genuine versus faked by the agent itself. Tier 1 rules derive from externally observable proofs the agent cannot falsify, such as valid JSON, file existence on disk, compilation success, test completion within timeout, non-empty output, and serving as ground truth without subjective interpretation. Tier 2 rules offer statistical signals compared to baselines not authored by the agent, such as embedding similarity, length checks, and diff impact assessments. Tier 3 rules rely on the model's judgment and are inherently non-deterministic, metered, and slow, making them suitable only for offline analysis, not real-time gating or regression testing.\n\nThe practical solution is to prioritize the first two tiers, which are deterministic, cost-free, and fast enough to reside in the hot path, blocking runs when violated. Tier 3 should remain offline, offering only opinion-based insights. By strictly testing the deterministic Tier 1+2 rules before integration, you mitigate the risk of untested, potentially flawed evals entering production.",
  "summary": "You wrote a new eval. It caught the failure you just saw in production. You shipped it as a gate. Congratulations — you now have a piece of untested code sitting on the hot path of every agent run, deciding what ships and what gets blocked. We treat agent evals as if writing them is the hard part. It isn't. The hard part is knowing your eval actually discriminates: that it goes red on the bad…",
  "key_points": [
    "Newly created eval rule detected production failure.",
    "False-negative gate from regex bug in rule.",
    "Tier 1 rules should be deterministic and hot path."
  ],
  "editors_take": null,
  "illustration": "https://urgent.news/ill/17134.png",
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}