TTS That Changes "Recording Room" Every Time You Generate — Style Falls Apart Unless You Match the Acoustics of Your Source Material

Generated TTS voices change their sound quality every time they are generated, even when using the same settings, model, and server. The variation in audio quality is most noticeable when switching between different emotional styles, such as joy, sadness, and anger. This is because the training corpus was generated using different reference audio for each emotion, which results in varying frequency characteristics.

The stricter the quality gate, the flatter the readings survive, leading to slightly different spectral shapes in each anchor. As a result, the TTS model learns not just the style but also the acoustic characteristics of the reference, causing the sound quality to shift between emotional groups.

To address this issue, a technique called LTAS (Long-Term Average Spectrum) was used to align all clips to a common frequency characteristic before training. The process involves computing the LTAS from the entire corpus or a reference clip group, calculating the difference between each clip's LTAS and the reference, and applying EQ to cancel out the difference.

The RMS gain is then normalized to prevent amplifying noise or introducing hiss in high-frequency ranges. Additionally, zero-phase processing is used to avoid shifting the phase during EQ application.

The main goal of this normalization process was to eliminate the sudden change in sound quality when switching emotional styles. However, an unexpected side effect was that extreme performance clips, such as screams or laughter, became trainable. Before normalization, these clips would degrade the entire model due to their high dynamic range and distinct spectra. After normalization, the model started learning the temporal structure of extreme performances, such as rapid pitch sweeps in screams.

While normalization helps remove irrelevant information, such as channel characteristics, overdoing it can cause issues like losing speaker identity or emotional acoustic features. The ±10dB correction limit helps maintain emotional differences while avoiding the loss of important information. Choosing the right reference for alignment is also crucial—using the average of the entire corpus can be problematic if many clips are of poor quality. To implement this technique, all clips should be processed in bulk just before training.