It agreed with the reference 100% of the time. It was right 75% of the time.
Swap a cheap model in behind an expensive one and the obvious way to check it is shadow traffic: send the same request to both, compare the answers, count how often they agree. High agreement, ship it. I measured that on 60 live calls while building SuperRouter. The routed model agreed with the reference 100% of the time . It was correct 75% of the time . Both numbers are real. The gap between…
When comparing two models, agreement does not guarantee correctness. 100% agreement between a cheap model and an expensive one could indicate both models are accurate, or the cheap model is flawed in the same ways as the expensive one. To determine correctness, evaluate the models against ground truth, which is generated, not manually labeled.
Planting known defects in the data allows testing of each model's ability to identify and correct errors. Both the success rate and the accuracy of the models should be measured. The ranking of models based on performance on easy cases should not be compared across datasets with different numbers of test cases. A leaderboard can indicate which models tend to perform well in general, but it cannot determine which model will perform best specifically for your use case.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
