A benchmark is only as good as the model you use to grade it

A benchmark's worth depends on the model used to grade it. The author built a pytest harness to test five language models, including a free local Llama, alongside GPT, DeepSeek, and two Claude models. The goal was to compare these models based on cost per query, speed, and answer quality. However, the author eventually stopped trusting the scoring tool and discovered that the paid models scored about the same as each other in terms of quality.

The scores ranged between 0.92 and 0.97 for all models, with the cheapest model scoring right in the middle. The most expensive model, Sonnet, scored no higher than the others, despite having longer answers that increased cost and latency. The author then checked the reliability of the scores by adding a range around each one and comparing the differences between pairs of models. This revealed that the gaps between the top models were due to noise rather than real differences.

The author questioned the judges, who were the free local Llama and the paid models. By re-grading the same answers with a paid judge, the author found that the paid models performed consistently, while the Llama underperformed. This inconsistency was attributed to the Llama being too soft on wrong answers, particularly in harder questions. The author conducted additional tests to demonstrate that the cheap judge was more lenient in its scoring, sometimes even passing wrong answers.

In conclusion, the benchmark's findings show that the paid models perform equally well in terms of quality, and the choice between them should be based on cost and speed rather than the score. The author emphasizes that blindly trusting the scoring tool can lead to inaccurate conclusions and highlights the importance of using a strict judge to catch a weak model's mistakes.