{
  "id": 7262484,
  "title": "AI Agent Testing: Why a 77% Pass Rate Can Mean 53% in Production",
  "url": "https://urgent.news/2026/09/14/ai-agent-testing-why-a-77-pass-rate-can-mean-53-in-production",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-14T06:40:20.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/abyzgenic/ai-agent-testing-why-a-77-pass-rate-can-mean-53-in-production-3ln"
  },
  "original_language": "en",
  "account": "A recent paper from IBM Research highlights the importance of consistency in AI agent performance. The paper, titled \"Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course,\" measures the consistency gap by running a ReAct agent on the AppWorld benchmark and executing each task five times. The authors found that while the average pass rate per run was 77%, the agent only passed all five runs 53% of the time. This 24-point gap is significant because a human can tolerate a system that fails 23% of inputs, but not one that handles the same input differently on different days.\n\nThe root cause of the inconsistency lies in the amplification of tiny differences during the agent's decision-making process. Factors like batching, hardware variations, and model updates can lead to different tool calls and observations, ultimately causing the agent to take a different path and arrive at a different result. The paper addresses this issue by building a Consistency Analyzer that identifies the specific step where the trajectories diverge and a Guideline Generator that writes targeted instructions for that step. These guidelines are then stored as episodic memory, improving the agent's performance on similar tasks.\n\nHowever, the authors do not claim to completely eliminate the consistency gap; they only narrow it. In practical applications, the required consistency may vary based on the volume of tasks processed. For instance, if an agent runs 400 times a day and a 2% flip rate results in eight incorrect vendor assignments, the consistency gap should be set higher to account for the higher volume of tasks. The authors suggest using N from your volume rather than the paper's value of five runs.\n\nThe reporter has adapted their testing process based on the insights from the paper. They now run each evaluation case N times and report both the per-run pass rate and the all-N pass rate. The per-run rate is presented as a footnote, highlighting the difference between the two metrics and the potential support tickets that could result from a lower all-N pass rate. Additionally, flaky cases, those that pass only a portion of the runs, are blocked from release, ensuring that the agent performs deterministically and accurately. Finally, the reporter emphasizes that deterministic steps should be made deterministic, or whole paths should be approved by a human if necessary.",
  "summary": "Short version for the impatient: if your agent passes 77% of your test cases, the chance it passes the same case five times in a row might be closer to 53%. That's the number I want you to carry around. If you want to know where it comes from and what I changed in my own testing because of it, read on. I ran into this the embarrassing way. Last spring I shipped an invoice-triage agent for a…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}