Measuring benchmark optimization in speech recognition

Public voice AI benchmarks have shown that models are performing at near-human levels. However, these scores do not always reflect how models function in real-world scenarios. Since public benchmarks are open and widely used, models may optimize for the tests themselves rather than improving in the underlying task. Traditional benchmarks often overlook many conditions that contribute to reliable, natural, contextually appropriate, and effective voice systems.

To address this, researchers introduced held-out sets in three different leaderboards: Real World VoiceEQ, Open-ASR, and Far-field ASR Leaderboard.

This phenomenon, known as benchmark optimization or "benchmaxxing," is difficult to measure in speech recognition. To quantify it, the researchers introduced three tests. They evaluated 11 widely used open-source Automatic Speech Recognition (ASR) models and found that several of the highest-scoring systems reproduced benchmark transcripts from VoxPopuli English and LibriSpeech datasets, even when the audio contradicted them, relevant words were silenced, or the audio supported two different written forms.

The researchers also developed consensus disagreement probes to test how leading ASR models handle transcription errors in benchmark datasets. They evaluated these models using an ensemble of independent models with low phoneme error rates. In some cases, models reproduced erroneous reference transcripts from benchmarks, such as omitting the phrase "Thank you" from a VoxPopuli clip. When presented with the same content in different voices, the error-favoring behavior often weakened or disappeared.

The results suggest that benchmark optimization is widespread and meaningful, with potential reference errors found in 40% of the VoxPopuli test clips analyzed, affecting about 3% of all reference words. Models exhibiting this behavior reproduced erroneous reference transcripts 18-30% of the time. The scatterplot compares Word Error Rate (WER) with the rate at which each model reproduces the incorrect reference transcript. The models with the lowest WER are also the most likely to reproduce these errors.