Urgent.News

What's breaking now, across thousands of outlets.

Tech

Conversation-Level vs Trace-Level Online Evals: Which Failures Each Can See

The unit you evaluate decides which failures you can see. A span-level evaluator judges one model call. A trace-level evaluator judges one request through the agent, including its tool calls. A conversation-level evaluator judges the whole multi-turn session after it ends. Most agent failures that cost customers only exist at the conversation level, but most eval tooling started at the span. This…

Online evaluation of conversational agents involves assessing models at different levels of granularity: span-level, trace-level, and conversation-level. Span-level evaluators examine one LLM call, trace-level evaluators assess one request including tool calls, and conversation-level evaluators judge the entire multi-turn session after it concludes. Most agent failures occur at the conversation level but online evaluation tools initially focus on span-level evaluations.

There are several types of failures that only become apparent through conversation-level evaluations. These include repetition and rephrasing, skipped prerequisites, contradictions across turns, inconsistent pricing, unresolved user goals, silent user abandonment, late false completions, and more. Conversation-level evaluators typically wait for user inactivity before judging to avoid premature false failures, but the default inactivity thresholds vary between tools.

To effectively use conversation-level online evaluations, the common failures seen at each evaluation level should be understood. Span-level evaluations are best for simple validity checks, trace-level evaluations detect tool-related issues and answer accuracy, while conversation-level evaluations expose multi-turn patterns like user repetition, skipped prerequisites, contradictions, unresolved goals or false successes.

When multiple failure types can be flagged at different levels, it's important to assign each failure to a single evaluation unit. Conversation-level verdicts should point to specific turns, and conversation rates should be reported per conversation rather than mixing denominators across units. Tools like Agnost run conversation-level evaluations by reading entire sessions against plain-English rules after a period of inactivity, while still tracking tool call metrics separately.

This approach captures the benefits of conversation-level context while maintaining the ability to pinpoint specific turn failures.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

More from Wednesday 18 November →