The Model Was Never the Problem
The shortlist post ended with a promise: stage two of the eval, selection within the shortlist, measured with a live model, scored so that a recall miss can never masquerade as a selection miss. This post pays that debt. The design came straight from the comment threads: same tool pools, same task pairs, shortlists from the same retriever, and every result row tagged with the description-set hash…
The headline, "The Model Was Never the Problem," highlights that the issue with a tool-calling system was not due to the underlying model. Instead, the design flaw was in the selection process. The study employed a shortlist of five tools generated through a deterministic shuffling method from the same retriever, ensuring that the correct tool and its four strongest BM25 distractors were always included.
The model used was gpt-5.4-mini, a small and inexpensive variant of the GPT-5.4 model. The study tested the model's performance using various descriptions and paraphrases, measuring selection accuracy across different conditions. The results showed that the model picked the right tool 90 to 97 percent of the time, regardless of the description text style or the presence of paraphrases.
The study also found that selecting the correct tool while filling in arguments for it was nearly flawless, even when the model had to work with paraphrased user inputs. This research emphasizes the importance of focusing on recall, or the ability of the system to retrieve the correct tool, before considering the costs and benefits of increasing model size.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
