{
  "id": 281009,
  "title": "Observability for Agents: Tracing a 40-Step Run",
  "url": "https://urgent.news/2026/08/07/observability-for-agents-tracing-a-40-step-run",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-07T21:20:57.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/multigrid/observability-for-agents-tracing-a-40-step-run-kea"
  },
  "original_language": "en",
  "account": "A forty-step process that went awry is not a single incident, but rather forty individual decisions – one or two of which were incorrect. Logging the initial request and final response provides only the starting point and endpoint. Debugging occurs in the intervals between these points. The issue with solely relying on request logging is that it fails to address cross-step questions: determining at which step things went awry, what was observed just prior to that, why a specific tool was invoked multiple times, where the eleven dollars were spent, and whether the failure matches previous occurrences.\n\nLogging alone cannot answer these inquiries, but a comprehensive trace can. While inventing a custom schema is unwise, OpenTelemetry offers a suitable solution. Its GenAI semantic conventions include fields such as gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.tool.name, which enable readable traces regardless of the chosen tooling. Moreover, agent spans are integrated into the same trace as the initial HTTP request, offering greater value than any agent-specific dashboard.\n\nThe span tree structure reveals the following key details: the agent.run task, run identifier, halt reason, and total cost. Each step is represented as a span with its index, gen_ai.chat model, token count, completion reason, latency, and the agent.tool list, hash, byte count, and error class. Step 39 specifically identifies the gen_ai.chat finish_reason as 'tool_calls,' which indicates where the failure occurred. This layout consists of four critical properties that greatly surpass mere naming.\n\nThe step span is not an event but a genuine span, allowing for aggregatable metrics like 'p95 step latency by tool.' Parallel tool calls are depicted as siblings under a single step, highlighting any unintended concurrency. Compaction is treated as a span due to its paid, lossy nature and because it is the first event to examine when an agent loses information. Every span carries the run identifier, ensuring that sub-agent spans merge with the parent trace instead of forming an orphan that requires timestamp-based correlation.\n\nTo ensure the accuracy of traces, the exact messages array should be referenced and stored in object storage keyed by a content hash. This hash acts as a deduplicator, sharing identical prefixes across runs within the same blob. Tokens per step (both directions) provide a diagnostic chart directly correlated with step index, revealing when a tool returns an unexpectedly large output. Cumulative cost on the root span avoids queries to sum costs later, as the run unit directly correlates with budgeting requirements. The halt reason is presented as a first-class attribute, offering a distribution over time that acts as a health metric for the agent.\n\nArguments and tool outputs are hashed and stored by reference, minimizing the risk of exposing sensitive user data. Raw arguments for tools handling credentials or personal information are stored as hashes, with the corresponding blob available for reference. Tool output is also hashed, complete with a bytes attribute to pinpoint tools consuming excessive context. Retry and provider metadata, such as attempt number, provider request ID, and cache usage status, help distinguish between similar runs executed at different times.\n\nTo prevent overloading your metrics backend, refrain from placing prompt content in span names or metric labels due to unbounded cardinality risks. Instead, store raw tool arguments as hashes with accompanying blobs for access control. Tail sampling should not be used, as it is ineffective at determining whether a run will be valuable. Instead, employ tail sampling with a predefined keep list: retain runs with non-finish halt reasons and those containing tool errors. Additionally, maintain a small uniform sample of successful runs (e.g., one in a hundred) to establish a baseline for comparisons. By focusing on these retained traces, you can distinguish between normal operations and occasional failures, preserving a clear understanding of \"normal\" behavior.",
  "summary": "A forty-step run that went wrong is not one incident, it is forty decisions of which one or two were bad. Logging the request and the response gives you the first and the last. The debugging happens entirely in between. Why request logging is not enough The questions people actually arrive with are all cross-step: at which step did it stop making sense, what did it see just before that, why did…",
  "key_points": [],
  "editors_take": null,
  "illustration": "https://urgent.news/ill/281009.png",
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}