Urgent.News

What's breaking now, across thousands of outlets.

AI

Your AI Agent Returned 200 OK and Still Got It Wrong: A Developer's Guide to Tracing Agents

Here's a bug report you'll eventually get if you ship an AI agent: "The refund agent approved a refund for a 7-week-old order. Policy is 30 days. Customer was thrilled. Finance was not." You open your dashboards. Error rate: normal. p95 latency: normal. HTTP status: 200 . The final message to the customer: polite, on-brand, perfectly formatted. Everything is green, and the agent still did the…

Agent observability is the ability to reconstruct the entire journey of an AI agent after the fact, including what the agent was asked, what it saw, what decisions it made at each step, which tools it called with which arguments, and what came back. This is achieved through four signals: traces, metrics, logs, and evaluations. Evaluations are the AI-specific addition, as an agent can return a 200 status code and finish quickly, yet still be wrong.

Agent failures often occur silently, with errors compounding across steps. Even if each individual step is successful 95% of the time, the end-to-end success rate can drop significantly (e.g., 95% each for 10 steps becomes only 60% overall). Agents can also fail non-deterministically, making them difficult to reproduce. Additionally, success codes like 200 OK do not guarantee the agent performed correctly, and costs can accumulate without notice.

To instrument an agent, use OpenTelemetry's GenAI semantic conventions, which include dedicated operations for agent work such as create_agent, invoke_agent, and execute_tool. This allows you to capture all the necessary information without reinventing a trace format. Use the opentelemetry-sdk, opentelemetry-exporter-otlp, and opentelemetry-trace libraries to set up telemetry and tracer providers, and start spans for each step of the agent's run.

By following these steps, you can effectively debug any failed traces and gain a comprehensive understanding of your agent's behavior.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in AI

Catching Silent AI Agent Failures in Production: Metrics, Drift and Hallucinated Actions

In Part 1 we instrumented an AI agent with OpenTelemetry and learned to read a trace: one run, broken into spans, showing exactly where a refund agent skipped a failed policy check.

  • Establish metrics for task success, quality, and cost to monitor AI agents
  • Set up alerts for success rate drop, looping agents, repeated tool usage
  • Detect agent drift by comparing behavior before and after changes

More from Thursday 19 November →