{
  "id": 9413495,
  "title": "How to tell if your AI agent actually got better",
  "url": "https://urgent.news/2026/09/23/how-to-tell-if-your-ai-agent-actually-got-better",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-23T20:40:15.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/apovalov/how-to-tell-if-your-ai-agent-actually-got-better-2p12"
  },
  "original_language": "en",
  "account": "Determining if an AI agent has improved is a common question. The answers to this query are simple but may not seem so at first. The agent’s output of \"You're booked for Thursday\" reveals nothing about its performance. Anthropic’s guidelines on agent evaluations indicate that the focus should be on the state the agent leaves behind, rather than the language it uses. The crucial questions to ask are whether the target row was inserted into the database, and whether the appropriate tool calls were executed with the right parameters. Demos often appear impressive because they showcase the agent’s replies, but an evaluation truly monitors the system’s behavior. While language models (LLMs) can help in some cases, they introduce their own biases. A study of 21 LLM judges demonstrated that their verdicts varied significantly, sometimes flipping their decision up to 38% of the time. Observing replies alone does not measure the agent's capability; it merely reflects its politeness. To accurately assess an agent's performance, it is necessary to calibrate the judge against human labels. For evaluation purposes, choosing between a 1 to 10 scale or a binary pass/fail system is recommended. Starting with 20 to 50 tasks, ideally drawn from real failures, proves more effective than relying on synthetic cases. Evaluating an LLM’s capabilities can sometimes precede actual use, as it establishes clear expectations. However, when addressing failures, the approach should be reverse: ship the product, analyze the traces, and then formulate an evaluation tailored to the specific failure encountered.",
  "summary": "People keep asking me how you actually know an agent got better. I collected my answers to the most common questions in one place. Isn't it obvious from the answers? No. The agent says \"You're booked for Thursday,\" and that sentence tells you nothing. Anthropic's guide on agent evals shows it with a flight-booking agent: it may say \"Your flight has been booked,\" but the outcome is whether a…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}