{
  "id": 6607605,
  "title": "Agent Evaluation Metric for multi-turn conversations",
  "url": "https://urgent.news/2026/09/10/agent-evaluation-metric-for-multi-turn-conversations",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-10T15:55:41.000Z",
  "source": {
    "name": "AWS Machine Learning",
    "slug": "aws-machine-learning",
    "url": "https://aws.amazon.com/blogs/machine-learning/agent-evaluation-metric-for-multi-turn-conversations/"
  },
  "original_language": "en",
  "account": "The Agent Evaluation Metric (AEM) offers a new way to assess multi-turn agent performance. Traditional holistic scores fail to pinpoint the exact turn where errors occur, especially when a single wrong decision cascades into subsequent turns. AEM tackles this problem by breaking down agent quality into named, measurable sub-metrics evaluated turn by turn.\n\nCorrectness is the first dimension implemented in AEM, with two sub-metrics: Truthfulness and Completeness. Truthfulness checks if the agent's factual outputs align with expectations, while Completeness verifies that all required elements, such as necessary parameters in tool calls or requested information in responses, are present. These sub-metrics are applied independently to each turn, whether it's a response or an action turn.\n\nThe AEM framework further decomposes correctness into structural checks (e.g., selecting the right tool) and content checks (e.g., factual accuracy and parameter correctness). Each turn undergoes this turn-level evaluation, and the results are composited into a single indicator score. This approach allows for extensibility to new evaluation dimensions such as safety, instruction retention, and reasoning depth without restructuring the evaluation process.",
  "summary": "Multi-turn agents fail in ways single-turn evaluation misses: one early mistake corrupts every later turn. This post introduces the Agent Evaluation Metric (AEM), a decomposable, turn-level way to measure agent quality, applied to its first dimension, correctness, to pinpoint the turn that caused a failure and separate it from the turns that inherited it.",
  "key_points": [
    "Agent Evaluation Metric (AEM) assesses multi-turn agent performance",
    "AEM breaks down agent quality into named, measurable sub-metrics",
    "Correctness sub-metrics: Truthfulness and Completeness evaluated turn by turn"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}