What Is an MCP Eval? Why Your Server Passes Every Test and Still Fails
📖 TL;DR An MCP eval is a realistic task a model must complete using only your server's tools — not an assertion about one call. A test asks "did the call work?". An eval asks "could an agent get the right answer?" The signature failure is an answer that is wrong even though every call returned 200. No inspector catches it. There are four useful outcomes : pass, wrong answer, too many calls, and…
An MCP eval is a realistic task a model must complete using only the server's tools, not an assertion about a single call. A test focuses on whether the call worked, while an eval assesses whether an agent can obtain the correct answer. The main failure scenario is when the model provides an incorrect answer despite all calls returning 200.
There are four outcomes: pass, wrong answer, too many calls, and untestable. Untestable is not a failure and should not be counted as such. Evaluating an MCP server is different from testing protocol correctness or inspecting for handshakes and tool functionality. An eval focuses on whether an agent can obtain the right answer from the server's descriptions.
When building an eval engine for arbitrary MCP servers, it's crucial to distinguish between pass, wrong answer, too many calls, and untestable outcomes to gain valuable insights.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
