Capability Parity Is Not an Eval
There's a controlled study going around that does something I wish more eval write-ups did: it holds the action contract fixed. Same executable capabilities on both sides — an MCP agent on one, a CLI agent on the other — and the behavior still split. That shouldn't be surprising. The fact that it is tells you something about how we've been comparing interfaces. The default comparison goes like…
A recent controlled study compares two different agent interfaces, MCP and CLI, in a way that's rarely seen in evaluation papers. The key insight is that simply counting the number of tools doesn't tell you anything about how well the model performs. The study holds the action contract constant by using the same executable capabilities on both sides, despite the fact that they are accessed differently.
The study measures four key differences between the two interfaces: discovery, error shape, composition, and irreversibility. These differences are important because they can impact the user experience and the overall reliability of the system. The study also highlights the importance of treating the interface as a variable when comparing models, rather than just reporting a single success rate.
The authors provide a method for running a behavioral parity suite, which measures divergence across multiple runs of the same task. By doing this, teams can make informed decisions about which interface to use based on factors other than just model performance.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
