{
  "id": 20564947,
  "title": "Conversation-Level vs Trace-Level Online Evals: Which Failures Each Can See",
  "url": "https://urgent.news/2026/11/18/conversation-level-vs-trace-level-online-evals-which-failures-each",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-18T08:01:19.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/shubhampalriwala/conversation-level-vs-trace-level-online-evals-which-failures-each-can-see-5d6e"
  },
  "original_language": "en",
  "account": "Online evaluation of conversational agents involves assessing models at different levels of granularity: span-level, trace-level, and conversation-level. Span-level evaluators examine one LLM call, trace-level evaluators assess one request including tool calls, and conversation-level evaluators judge the entire multi-turn session after it concludes. Most agent failures occur at the conversation level but online evaluation tools initially focus on span-level evaluations.\n\nThere are several types of failures that only become apparent through conversation-level evaluations. These include repetition and rephrasing, skipped prerequisites, contradictions across turns, inconsistent pricing, unresolved user goals, silent user abandonment, late false completions, and more. Conversation-level evaluators typically wait for user inactivity before judging to avoid premature false failures, but the default inactivity thresholds vary between tools.\n\nTo effectively use conversation-level online evaluations, the common failures seen at each evaluation level should be understood. Span-level evaluations are best for simple validity checks, trace-level evaluations detect tool-related issues and answer accuracy, while conversation-level evaluations expose multi-turn patterns like user repetition, skipped prerequisites, contradictions, unresolved goals or false successes.\n\nWhen multiple failure types can be flagged at different levels, it's important to assign each failure to a single evaluation unit. Conversation-level verdicts should point to specific turns, and conversation rates should be reported per conversation rather than mixing denominators across units. Tools like Agnost run conversation-level evaluations by reading entire sessions against plain-English rules after a period of inactivity, while still tracking tool call metrics separately. This approach captures the benefits of conversation-level context while maintaining the ability to pinpoint specific turn failures.",
  "summary": "The unit you evaluate decides which failures you can see. A span-level evaluator judges one model call. A trace-level evaluator judges one request through the agent, including its tool calls. A conversation-level evaluator judges the whole multi-turn session after it ends. Most agent failures that cost customers only exist at the conversation level, but most eval tooling started at the span. This…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}