Agent Evaluation Metric for multi-turn conversations
Multi-turn agents fail in ways single-turn evaluation misses: one early mistake corrupts every later turn. This post introduces the Agent Evaluation Metric (AEM), a decomposable, turn-level way to measure agent quality, applied to its first dimension, correctness, to pinpoint the turn that caused a failure and separate it from the turns that inherited it.
The Agent Evaluation Metric (AEM) offers a new way to assess multi-turn agent performance. Traditional holistic scores fail to pinpoint the exact turn where errors occur, especially when a single wrong decision cascades into subsequent turns. AEM tackles this problem by breaking down agent quality into named, measurable sub-metrics evaluated turn by turn.
Correctness is the first dimension implemented in AEM, with two sub-metrics: Truthfulness and Completeness. Truthfulness checks if the agent's factual outputs align with expectations, while Completeness verifies that all required elements, such as necessary parameters in tool calls or requested information in responses, are present. These sub-metrics are applied independently to each turn, whether it's a response or an action turn.
The AEM framework further decomposes correctness into structural checks (e.g., selecting the right tool) and content checks (e.g., factual accuracy and parameter correctness). Each turn undergoes this turn-level evaluation, and the results are composited into a single indicator score. This approach allows for extensibility to new evaluation dimensions such as safety, instruction retention, and reasoning depth without restructuring the evaluation process.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.