A single rough clip can ruin the entire style — How to choose 5 good ones
📝 Originally published (in Japanese) at forge.workstyle.tech . The TTS we're using creates emotional styles from "a few representative clips." For a joyful style, for example, passing a few joyful audio clips registers an average style vector. At first, I simply passed the first five clips from each group: clips = corpus_clips [ emotion ][: 5 ] register_style ( model_id , style_name = emotion ,…
A single rough clip can significantly ruin the overall style when synthesizing audio. To create joyful or other emotional styles, the system uses five representative audio clips to register the average style vector. However, if one of those five clips is rough, it can negatively impact the entire synthesized audio. The issue arises because rough audio often lies far outside the normal range in the direction of hoarseness, which, when averaged with normal clips, pulls the center of gravity toward the outlier. Consequently, all audio synthesized with that style carries a hoarse quality.
Roughness in a clip can stem from two primary factors: jitter (period fluctuation) and octave jumps. Jitter refers to unstable vocal cord vibration periods, resulting in a rough impression. Octave jumps occur when the fundamental frequency (F0) estimation jumps to double or half between adjacent frames, which can result from actual voice cracking or estimator errors. Both jitter and octave jumps manifest as unstable sound.
To identify rough clips, the system measures jitter and octave jump rate and selects clips with the lowest values. A function called stability_score(wav_bytes) returns (jitter, octave_jump_rate), with lower values indicating more stability. The top n most stable clips are then chosen using the stable_top(clips, n = 5) function, which weighs octave jumps more heavily because they have a greater impact on perception.
However, there are limitations to the stability_score function. It overestimates jitter in lower voices due to errors in F0 estimation, leading to uniformly disadvantaged lower voices in comparisons. Additionally, the metric cannot be used for cross-speaker comparisons, especially between genders. Therefore, two separate quality gates are necessary: one for correct content and another for clean sound.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
