Coverage Is Not Capability: What Whisper Taught Me About African Speech

When I initially built the first version of Scribe AI in July 2026, I assumed speech recognition would be a straightforward component. OpenAI's Whisper was open-source, widely used, and marketed as multilingual. I integrated it into the live pipeline and proceeded to tackle the more complex aspects. However, my expectations were not met when I encountered African languages and accents that proved difficult for the model.

The performance was unreliable enough to not serve as a foundation for further development. The issue did not stem from a problem in my pipeline; rather, it aligned with the inherent limitations of Whisper's training data.

Whisper was trained on 680,000 hours of labeled audio. The dataset distribution shows 438,218 hours of English speech recognition, 125,739 hours of speech translation into English, and 117,113 hours of speech recognition across 96 other languages. Within this latter group, the distribution of language coverage is highly uneven. For instance, Swahili, one of Africa's most widely spoken languages, has only approximately 5.4 hours of transcribed speech, while Dholuo, another important African language, is not even listed in Whisper's language list.

Radford et al., the authors of the Whisper paper, note that the amount of pre-training speech recognition data for a language is a strong predictor of its zero-shot performance on that language.

This mismatch between the training data and the actual speech needs was my primary mistake. I had viewed multilingual capabilities as a uniform property of the model, failing to recognize the significant variations in coverage and performance across languages. To address this, it's crucial to evaluate a model's capability based on actual usage scenarios rather than relying solely on its supported-language list. Three key questions should guide this evaluation process:

1. Does It Work on My Speakers? Real-world audio can differ significantly from clean, benchmark recordings. Factors such as spontaneous speech, unfamiliar accents, background noise, and rapid language switching can challenge a model's performance. To gain insight into how well a model operates with your specific use case, it's essential to test it on a small set of representative recordings, ideally transcribed by hand to establish a reference standard.

The focus should shift from mere headline metrics to an understanding of the model's actual performance on the intended audio environment.

2. Can I Use It for What I Am Building? Model quality is important, but it's not the only consideration. Licensing also plays a crucial role. While the MMS models were initially released for non-commercial use only, this may not be suitable for commercial applications. When selecting a model, consider the licensing terms along with performance metrics, intended use, redistribution conditions, and deployment requirements.

Ignoring these factors can lead to significant obstacles in implementing the model in your desired application.

3. Can It Run Where My Users Are? The environment in which the model operates is as critical as its accuracy. For Scribe AI, which is meant to run on low-resource hardware like laptops typically found in churches, factors such as model size, speed, memory requirements, quantisation support, and CPU performance become paramount. A model might achieve excellent accuracy in a data center setting but fail to keep up with live speech on the target device.

Therefore, assessing the model's inference speed, memory usage, support for quantisation, and CPU performance is essential to ensure it can meet the demands of the actual deployment scenario.

These lessons underscore that speech recognition is not merely about checking which languages a model supports but understanding the broader context of its performance. For Scribe AI, the solution involved designing the system with modularity in mind, allowing for easier integration of a more suitable speech recognition model in the future.

Additionally, incorporating a human verification step ensures that even imperfect recognition can still be useful. Looking ahead, developers aiming to build voice products for speakers of African languages, such as Dholuo, Kalenjin, or Kenyan-accented English, need to carefully evaluate their options based on these criteria. While models like Meta's Massively Multilingual Speech (MMS) and Omnilingual ASR offer broader coverage, their performance varies significantly across languages, particularly low-resource ones.

Achieving reliable speech recognition in these contexts requires rigorous evaluation, consideration of licensing impacts, and a clear understanding of the deployment environment.