Your TTS Model Sounds Great — Until It Says "GPUB"
Originally published at ai.bedvibe.studio . I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it. Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the…
I developed a text-to-speech (TTS) product, but it consistently struggled with certain inputs like numbers, dates, acronyms, names, URLs, and even single letters. The typical metric used to assess TTS quality, Word Error Rate (WER), failed to accurately reflect the issues. It would flag perfectly good audio as broken when the script mentioned times like "3:30 PM" and would miss real failures on short tokens, where the speech recognition was just as unreliable as the TTS itself.
To address this, I created TTSproof, a QA framework that tackles these problems. It includes a set of structural checks that can identify audio quality issues without needing a model. These checks look for empty or truncated audio, duration explosions, long internal silences, clipping, loops, and end-of-clip artifacts. Additionally, TTSproof uses a form of WER/CER that normalizes the expected text and the automatic speech recognition (ASR) transcript to spoken form, so errors like "3:30 PM" versus "three thirty" are no longer counted as mistakes.
TTSproof also incorporates an ASR-uncertainty quarantine, where the system sets aside short utterances if the ASR is unsure, and has a human review them instead of auto-failing the sample. In a study of 390 samples using three different voices, no structural audio-integrity defects were found, meaning that any failures were indeed pronunciation issues. Out of 42 uncertain clips, 55% were ASR false negatives (the TTS got it right, but the recognition failed) and 45% were genuine TTS mispronunciations.
In addition to the structural checks, TTSproof includes a corpus of 817 curated edge cases covering various categories such as numbers, dates, proper names, and more. This corpus is versioned independently from the software, ensuring consistent scores even as tools are updated. The framework can be used to benchmark any TTS engine in a single command, providing a detailed report with waveforms, audio playback, the actual ASR output, and a CI regression gate.
The open-source nature of TTSproof allows for community contributions, which have already led to real bug fixes in the system.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
