LLM observability: tracing, monitoring, and debugging agents in production

LLM observability refers to the ability to gain insights into the runtime behavior of large language models (LLMs) or agent systems. This includes traces, metrics, and logs that provide visibility into what a model and its agent loop did during each request, enabling the diagnosis of failures in production. A trace records each step of the process, such as model calls, tool calls, and retrievals, along with their inputs, outputs, token usage, latency, and cost.

This information helps answer what happened during a specific run, whereas evaluation measures the correctness of outputs across a dataset. To achieve observability, you need the three signals: traces, metrics, and logs. Traces show the full execution path of a request as a tree of spans, capturing model calls, tool calls, and retrievals.

Metrics provide numeric aggregates over many requests, such as token usage, latency, cost, error rate, and throughput. Logs contain structured payloads like prompts, completions, tool arguments, results, and events. LLM observability is essential for debugging issues in production, where non-deterministic behavior makes reproducing failures locally challenging.

It is a crucial component of the AI quality stack, which includes evaluation and monitoring. OpenTelemetry, an open standard for traces, metrics, and logs, is commonly used for LLM observability, ensuring portability and compatibility across different tools and systems.