{
  "id": 7115634,
  "title": "It passed CI. It passed your evals. The customer still got the wrong answer.",
  "url": "https://urgent.news/2026/09/13/it-passed-ci-it-passed-your-evals-the-customer-still-got-the-wrong",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-13T14:00:00.000Z",
  "source": {
    "name": "The New Stack",
    "slug": "the-new-stack",
    "url": "https://thenewstack.io/ai-agent-trace-debugging/"
  },
  "original_language": "en",
  "account": "Dynatrace's 2026 State of SRE and Platform Engineering report reveals an observability gap when it comes to AI features. While 77% of platform engineering teams embed observability in at least some services, only 40% have full integration across all deployments. This gap becomes a liability with AI agents, as they fail quietly, returning a 200 status code and passing faithfulness checks even when the customer receives incorrect answers.\n\nIn a case where a support agent uses product documentation to answer a customer's question about exporting in version 2026.3, the documentation lookup becomes slower after the release. By comparing one run affected by the change with a similar run before the change, it becomes evident that the issue lies in the model's repeated searches for the same document version. The trace shows three identical searches, each taking a few seconds, while the actual retrieval of the relevant document version is the longest part of the trace, taking 8.1 seconds.\n\nThe root cause appears to be a lack of version metadata in the documents used for lookup. When the system fails to provide the correct version in response to the customer's request, it results in the agent providing inaccurate information. To address this issue, it is essential to ensure that the system includes version metadata in the retrieved documents and assert on the lookup directly, without relying on the model. This approach will help prevent similar issues from arising in the future.",
  "summary": "A diff is not evidence. It’s a statement of intent. The tests passed. The review’s done. The change is live. The post It passed CI. It passed your evals. The customer still got the wrong answer. appeared first on The New Stack .",
  "key_points": [
    "77% of platform engineering teams embed observability in services",
    "Only 40% have full integration across all deployments",
    "Lack of version metadata leads to inaccurate AI agent responses"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}