My LLM app was fully traced. During an incident the trace was still useless.

A fully traced LLM app was not actually helpful during an incident, despite having comprehensive tracing in place. The trace store presented a flat list of 28 spans, many of which lacked crucial information such as the prompt version, model-call details, and retrieval queries. This made it difficult to pinpoint the regression, which had caused a drop in quality for German enterprise users.

The rewritten tracing system aimed to answer key questions instantly, such as the prompt version used, the slowest retrieval, failed tool calls, quality score changes, affected cohort, and the model responsible for the response. A good trace should be structured as a tree, mirroring the actual user request and subsequent steps, rather than a flat list of spans.

Adding the prompt version to every model-call span as three tags—prompt, version, and A/B variant—proved to be the most valuable change, allowing for quick identification of the regression's cause. Including quality scores on spans, rather than just latency and errors, enabled better monitoring of the model's output quality. Separating token costs into reasoning and cache categories helped in identifying cost anomalies more effectively.

Redaction of sensitive data should occur at the collector level, ensuring compliance while maintaining traceability. Stable span names were essential for consistent dashboard and alert grouping. Lastly, focusing on tail traces instead of head traces during incidents proved more beneficial, as rare failures were less likely to be captured through random sampling.