{
  "id": 20564941,
  "title": "Your Agent Is Failing in Production Even When Your Evals Pass",
  "url": "https://urgent.news/2026/11/18/your-agent-is-failing-in-production-even-when-your-evals-pass",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-18T08:02:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/shubhampalriwala/your-agent-is-failing-in-production-even-when-your-evals-pass-3h3a"
  },
  "original_language": "en",
  "account": "An agent's passing evaluation score does not guarantee successful operation in real-world production. Passing evaluations indicate the agent handled a curated set of examples under controlled conditions, but production environments are far more complex. Users interact with agents using half-formed questions, change goals mid-thread, reference account state the model cannot see, or abruptly stop without explaining why.\n\nOffline evaluations, performed before release on fixed datasets, do not capture these production challenges. Online evaluations, which assess real-time production traffic, are needed to detect failures that offline evaluations could miss. Production failures often go unnoticed in offline evaluations because they are subtle and do not manifest as error messages. Instead, they appear as users who do not upgrade, abandon the product, file support tickets, or exhibit workflow friction that erodes trust.\n\nThe main reasons why evaluations pass while users fail are:\n1. Offline evaluations assume known inputs, success conditions, and stable context, which is rarely the case in production.\n2. Users often communicate in ways that differ from the agent's training data, using informal language and implicit goals.\n3. Production environments are dynamically changing, unlike the static data used in training evaluations.\n\nThe most common failure mode is silent non-resolution, where the agent's answer may be plausible but fails to achieve the user's goal, leading to churn or lost trust. Repeated friction from the same user hitting the same unresolved issue across sessions also contributes to trust degradation and lost revenue. Misreading buying intent is another issue, where the agent might handle informational questions about product features but fail to identify actual sales or expansion opportunities.\n\nTo properly assess agent reliability, it is essential to monitor production behavior rather than relying solely on evaluation pass rates. Key metrics to track include the share of conversations where the user's goal was not met, repeated unresolved issues from the same user, frustration levels linked to unmet goals, the rate of missed upgrade opportunities, and trends in post-change failures. By connecting production monitoring data with evaluation results, teams can better identify and address reliability issues before they impact user experience and business outcomes.",
  "summary": "Your eval suite passing is not the same thing as your agent working. It means the agent handled a curated set of examples under controlled conditions. That is useful. It is also much narrower than the world your users live in. Production is messy. Users ask half-formed questions, change goals mid-thread, reference account state the model cannot see, and disappear without telling you why. This is…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}