18 Insights from Mass-Producing Voice Models โ From Diffusion TTS Voice Design to Training Corpus Creation and Quality Gate Pitfalls
๐ Originally published (in Japanese) at forge.workstyle.tech . This is a record of designing voices from single-line captions, automatically creating a learning corpus, and passing all 12 role-specific voices (narrator/counselor/sales/presenter/operator/MC for both men and women) through full inspection. I wrote about the failures I encountered during approximately one month of actual work,โฆ
In this article, a report is presented detailing the process of mass-producing voice models, from design to operation. The report covers 18 insights gained during this endeavor.
The first chapter discusses the challenges of designing voices. The chosen TTS model was too slow for real-time conversation, leading to a decision to use a two-stage approach: initially designing voices with diffusion TTS, then utilizing pre-trained models for speech. Each voice is determined by its caption and a random seed, making it fully reproducible.
Chapter 2 focuses on the manufacturing aspect, where the quality of the corpus directly influences the voice produced. A stricter quality gate results in more monotonous voice takes, as emotions are often lost in the process. The report also discusses the difficulty of changing speech rate after training, as it is baked into the corpus and cannot be adjusted via synthesis parameters.
Chapter 3 explores the inspection process, highlighting the distinction between having a gate and it being effective. It reveals that six out of the 18 articles focus on ineffective gates. An example is provided where the AI's habit of stretching "Hello" was due to kana normalization during verification, rendering the defect undetectable. Additionally, the report addresses issues with broken input text, such as missing characters and hallucinations, and the limitations of STT-only inspection in capturing additive sounds.
Chapter 4 delves into the operation phase, where maintaining continuous training material is crucial. A network blink resulted in the loss of 70 minutes of training data, necessitating 205 retries. The report also discusses the impact of registration paths on asset management and the challenges of deploying workflows without proper version control. It emphasizes the importance of measuring the right metrics and not getting distracted by unmeasured targets.
Throughout the article, a common theme emerges: most failures stem from inadequate inspection design rather than model or GPU issues. The report concludes by suggesting that addressing unmeasured aspects is more effective than seeking smarter models. The insights gained from this experience are compiled in the "Manufacturing Pipeline for Practical Voices from Diffusion TTS."
Written by urgent.news from Dev.to's reporting โ not their text. Machine-written โ may contain errors; check the original before relying on it.