Why Standard WER Lies to You in Dialect AI

From my experience living in the United States, I discovered a significant flaw in standard voice agent architectures: they are biased towards American accents. When an input included non-standard accents, regional dialects, or code-switched phrasing, the system failed to process it correctly. This led to increased Word Error Rates (WER), as large language models expended more tokens attempting to understand the garbled text.

Having worked in a call center in the US, I witnessed firsthand how these systems falter due to their lack of audio perception. Once the automated ASR misinterprets an accented phrase, the downstream model blindly accepts the corrupted text as accurate, incorrectly guessing the caller's intent and failing to provide proper service.

This prompted me to establish Accent Labs with the goal of resolving the transcription bottleneck and enabling voice technology to effectively serve all individuals. When assessing speech recognition for low-resource languages and code-switched dialects, such as Nigerian English or West African Pidgin, standard WER fails to provide valuable insights.

Even when my ASR model transcribed speech accurately, it still received low WER scores. The standard WER calculation, which counts insertions, deletions, and substitutions against a ground-truth text, lacks semantic understanding. It flags every error as a formatting, punctuation, or orthographic shift, regardless of whether the model misunderstood the speaker's intent.

Normalizing hesitations, converting phonetic representations, and capitalizing words all contribute to substantial penalties, resulting in a WER increase from 0.11-0.22 to 0.333. This is not a sign of model degradation but rather an active cleaning process by the language model. To address this issue, I experimented with Gemma 4, a 26B MoE model (google/gemma-4-26b-a4b-it).

Its Mixture-of-Experts architecture allows for lower VRAM usage while maintaining low latency. Gemma 4's multi-token routing better preserves dialect markers (waka, dey, sha) compared to other SOTA models. By switching to Gemma 4, I reduced my WER from 0.333 to 0.333, outperforming Claude Haiku, which had a WER of 0.344. Claude Haiku's performance suffered due to its sensitivity to prompt constraints.

Over-prompting led to Haiku over-correcting valid dialect syntax into Standard English. The key to achieving near 0% WER on un-normalized ground truth lies in establishing an Audio Label ecosystem powered by human-in-the-loop data. I am developing Audio Label, a platform that facilitates a crowdsourced human review ecosystem for annotating raw audio and verified text pairs.

By designing the platform around a lightweight, link-based workflow, I aim to streamline the annotation process, enabling teams to continuously convert edge-case failures into proprietary training data and build an end-to-end speech model tailored to underrepresented dialects.