How to evaluate an AI router's 'none of these' branch
An AI router has to recognize requests outside its supported categories while still accepting useful work. Those are two different things to measure. Between 23 and 28 October 2026, I compared four decision models and five small language models. This article focuses on one task from that comparison: choosing a domain for a request, including an explicit "out of scope" option. The decision models…
To assess the performance of an AI router, it is essential to consider two distinct aspects: correctly declining out-of-scope requests and the rate of mistakenly declining in-scope requests. By focusing on these measures, alongside overall accuracy, a more comprehensive evaluation can be achieved.
For a router equipped with an out-of-scope option, three key metrics should be reported in addition to the overall accuracy. Out-of-scope recall quantifies the proportion of genuinely out-of-scope requests that were correctly identified. The false-rejection rate indicates the percentage of in-scope requests that were erroneously declined.
Lastly, the in-scope routing accuracy represents the proportion of requests correctly assigned to their respective domains, accounting for both declined and wrongly declined requests. In the given sample, the denominator for the first metric is 190, whereas the other two denominators are 810.
Mammoth-1 demonstrated an 80% out-of-scope recall, but its false-rejection rate amounted to 100 out of 810 in-scope requests. Its in-scope routing accuracy was reported as 83.7%. It is crucial to separate domain accuracy from the rejection count, as a valid request can still be sent to an incorrect destination. Additionally, tracking malformed answers and request failures separately is recommended.
In the benchmark, format failures were considered wrong in overall accuracy but were not excluded from the successful responses.
To apply this evaluation to your own router, first, meticulously identify the requests supported by each destination. Then, compile a labeled evaluation set containing both supported requests and plausible requests outside the scope. Ensure that the requests, category definitions, and option guidance remain consistent across models during the initial comparison.
Record the model versions and settings to facilitate future reference. After scoring, scrutinize examples from each error category. False rejections may indicate narrow option guidance, while missed out-of-scope requests may reveal unclear boundaries between supported and unsupported work. The specific consequences of these mistakes should be evaluated in the context of the application.
A confidence threshold can be experimented with by evaluating it on separate validation data and verifying its calibration. The reported counts reflect the options selected by the models but do not establish an optimal threshold. Finally, it is crucial to inspect the experiment using the provided public repository, which includes the harness, task definitions, raw requests and responses, and reports.
Start by examining the saved results and reports to review the existing experiment. The README explains how to rerun the experiment. Note that all nine models ran the CLINC task, and a production router should still be evaluated on its own traffic. Additionally, the comparison on the site covers calibration, format failures, latency, cost, and repeatability across other tasks.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
