In-Depth Explanation of the Seed-VC Architecture — Decomposing Voice into 'Who, What, and How' in a 4-Stage Structure

The Seed-VC architecture is a methodology for converting recorded voice into someone else's voice by breaking down the speech into three main components: who is speaking (speaker identity), what is being said (content), and how it's being said (prosody). This separation allows for independent manipulation of each component, resulting in accurate voice conversion without altering other aspects of the speech. The architecture employs four main modules to achieve this.

The first module, whisper, uses the encoder of the speech recognition model Whisper to extract content, which is represented as semantic features. This step involves resampling the input to 16kHz, as Whisper is designed for 16kHz audio. Whisper's limitation of processing up to 30 seconds at a time requires chunking for longer audio recordings. The output is a sequence of semantic features that represent the spoken content without pitch or timbre information.

The second module, campplus, condenses speaker identity into a 192-dimensional vector. This is achieved using the CAMPPlus speaker embedding model, which takes audio input, computes fbank features, and outputs a single vector. The speaker identity is crucial for the voice design app as vectors can be blended, added, or interpolated to create new voices.

This is accomplished by precomputing embeddings from clean recordings of 18 speakers as anchors and then blending them using weighted averages based on slider inputs defined by the user.

The third module, CFM/DiT, uses a diffusion model to generate a mel-spectrogram conditioned on the semantic features and speaker embedding. The diffusion model starts from random noise and gradually generates a mel-spectrogram. The mel-spectrogram is then trimmed to remove the frames corresponding to the original audio, leaving only the new content synthesised with the new speaker's voice.

This step integrates the outputs from the previous modules, with the semantic features representing the content, and the speaker embedding representing the identity.

The final module, BigVGAN, converts the mel-spectrogram back into an audible waveform. This completes the voice conversion process, resulting in a synthesized voice that sounds like the original speaker but with the desired content and prosody.