How to test whether a model can tell your MCP tools apart
You can read your own tool descriptions all day and not know whether a model can tell them apart. I spent a few weeks measuring 82,549 public MCP tool descriptions, and the most useful thing I learned is that the measurement has a ceiling. Here are two tools from a real server: score_resume "Score a resume for ATS compatibility." analyze_job_description "Extract what a job posting actually…
Testing a model's ability to distinguish between MCP tools involves a method that takes an afternoon and provides evidence for potential collisions. To begin, write two sentences for each pair of suspect tools: a control request that should route to one tool unambiguously, and a probe request that could reasonably route to either. Run each sentence through the model at a fixed temperature a few times, recording which tool is chosen in each instance.
First, examine the control sentences. If the control request is routed incorrectly, this indicates that the description for that tool is flawed and needs to be rewritten. If the control request is routed correctly, it serves as a gate to move on to the next pair of tools.
Next, analyze the probe sentences. Even if the probe consistently routes to one tool, it is still considered a pass, as the model has found a distinction that you may not have noticed. However, if the probe routes to different tools for the same input, this indicates a real collision, and the specific sentence serves as proof of this distinction.
When ranking pairs of tools to test, use static analysis as a filter to identify the most likely candidates. Focus on tools that share inputs and have distinct outputs, as they are more likely to exhibit collisions. Once the top fifty pairs are selected, probe them to confirm the presence of any splits.
There are three possible outcomes when a split is found: rewrite the descriptions for contrast, name the output instead of the input, or merge the tools into one with a mode parameter. Rewrite for contrast is the cheapest fix and does not alter the tool behavior. Renaming outputs helps differentiate between tools that have distinct outputs. Merging tools is appropriate when the split indicates that the tools have different intents and the user's request can be fulfilled by either.
Lastly, always keep the failures in mind. Store the prompt, expected tool, and the tool that was actually picked for each split. This information will be valuable when testing future revisions to the descriptions. By running the probe after any changes, you can verify whether the distinction between the tools has been closed, ensuring that the description modifications are effective and do not introduce new issues.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
