LLM Observability: Tracing, Logging, Debugging Agent Runs

In the world of AI, large language models (LLMs) have proven to be powerful tools, but they also come with their own set of challenges. One of the most pressing issues is the difficulty in understanding how these models behave in real-world scenarios. This lack of visibility often leads to silently failing LLM applications, as seen in a logistics client case where an agent was providing incorrect shipment status information to customers.

To address this problem, a careful approach to LLM observability is essential. Traditional observability tools assume that code is deterministic, meaning that errors can be traced back through stack traces. However, LLMs break this assumption in four critical ways. Firstly, the output of an LLM is nondeterministic, meaning that the same prompt can produce correct answers at different times, even without any code changes.

Secondly, the system's hidden state, consisting of the prompt, retrieved chunks, tool outputs, and model versions, plays a significant role in determining outcomes. Thirdly, costs are directly tied to the number of tokens processed, and any inefficiencies can quickly escalate expenses. Lastly, LLMs often provide confident but incorrect answers, which go unnoticed without proper logging and analysis.

To overcome these challenges, a three-pronged approach to observability is recommended: tracing, logging, and evaluations. Tracing provides insights into what happened, in what order, and how long each step took. This is achieved by creating a trace, which is a tree of spans representing the request, retrieval, prompt assembly, and each LLM call.

Each span carries crucial information such as duration, token counts, costs, model names, and input/output data. Logging, on the other hand, involves recording the full prompt and completion for every run, retrieved chunks with their scores and sources, tool arguments and results, and a stable run ID. This raw data is essential for audits, compliance, and reproducing specific failures.

Evaluations and metrics focus on determining whether the system is improving or deteriorating over time. Metrics include counters and gauges, such as tokens per request, latency percentiles, cost per resolved task, tool-call rate, and cache hit rate. Evaluations consist of scored test cases run against a regression set when changes are made to the prompt or model.

By analyzing the trace and evaluating the results, teams can determine whether changes have been successful and maintain the system's quality.

For example, consider an LLM agent tasked with retrieving shipment status and responding to customer inquiries. A trace of this agent's actions would reveal three primary failure points: retrieval returning a low-confidence chunk, a tool call returning a hard error, and the model answering incorrectly despite the error. By identifying these failure points, developers can focus their debugging efforts and improve the overall robustness of the system.