It passed CI. It passed your evals. The customer still got the wrong answer.
A diff is not evidence. It’s a statement of intent. The tests passed. The review’s done. The change is live. The post It passed CI. It passed your evals. The customer still got the wrong answer. appeared first on The New Stack .
Dynatrace's 2026 State of SRE and Platform Engineering report reveals an observability gap when it comes to AI features. While 77% of platform engineering teams embed observability in at least some services, only 40% have full integration across all deployments. This gap becomes a liability with AI agents, as they fail quietly, returning a 200 status code and passing faithfulness checks even when the customer receives incorrect answers.
In a case where a support agent uses product documentation to answer a customer's question about exporting in version 2026.3, the documentation lookup becomes slower after the release. By comparing one run affected by the change with a similar run before the change, it becomes evident that the issue lies in the model's repeated searches for the same document version.
The trace shows three identical searches, each taking a few seconds, while the actual retrieval of the relevant document version is the longest part of the trace, taking 8.1 seconds.
The root cause appears to be a lack of version metadata in the documents used for lookup. When the system fails to provide the correct version in response to the customer's request, it results in the agent providing inaccurate information. To address this issue, it is essential to ensure that the system includes version metadata in the retrieved documents and assert on the lookup directly, without relying on the model. This approach will help prevent similar issues from arising in the future.
Written by urgent.news from The New Stack's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
