A Closer Look at Phonon TTS. Launching High-Quality On-Device Speech That Tops the Benchmarks.
Two weeks ago, we introduced Phonon, our compact text-to-speech model that operates on your own device without relying on servers or networks. This privacy-focused approach has been well-received, prompting an in-depth examination of the launch. We will delve into the reasons behind Phonon's creation, its capabilities, the accompanying demos, and its performance on Hugging Face's Open TTS Leaderboard.
In contrast to most voice AI that operates in the cloud, Phonon processes your text on your device, ensuring that the voice works without an internet connection. We have also made the model's weights available under the Apache 2.0 license, permitting free usage, even in commercial products. In comparison, other models are licensed under CC BY-NC, which restricts usage to non-commercial projects. The commercial pricing for these models is determined by the number of users, rather than the amount of speech generated.
Creating a high-quality voice for small devices has historically been challenging. One strategy involves reducing model size, but this often results in subpar audio quality. Conversely, optimizing for quality can make the model too large for small devices. To address this dilemma, we have developed Phonon by collaborating with partners and early adopters. Over the past six months, we refined the model, reducing its size while adding support for more languages and custom voices through voice cloning and voice design.
Phonon is derived from the Pocket TTS lineage, employing flow matching to generate audio. This technique differentiates Phonon from other TTS systems and contributes to its exceptional quality despite its small size. However, training the model presented new challenges, including numerical instability across five languages. We overcame these obstacles through various training techniques, resulting in a high-performing model.
Phonon is available in two sizes: approximately 65 MB and 75 MB, both speaking English, French, German, Spanish, and Portuguese. The model includes over 100 built-in voices, with the option to create custom voices using Gradium Studio. Additionally, Phonon excels at handling specific tasks that other TTS systems often struggle with, such as interpreting phone numbers, dates, and codes.
Phonon is compatible with various devices, including phones, laptops, single-board computers, and browser tabs, enabling immediate audio playback.
During the launch, we showcased demos from our team and friends. On an Apple Watch with Wi-Fi disabled, Phonon guides users through a coffee brewing process. On an iPhone without a signal, it counts repetitions to maintain motivation. A robot named Lamp, developed by Autonomous, dons a sorting hat and converses in character. In a browser, an offline dino runner transforms into a radio show hosted by Rex.
Furthermore, a sci-fi game features voice acting powered by the player's device, and Proust reads aloud selected passages.
Hugging Face developed the Transformers integration, allowing developers to incorporate Phonon into their projects from the outset. The community has already built upon Phonon, with ExecuTorch enabling the model to run on a visitor's GPU within a browser, Micdrop integrating Phonon into Node.js applications, and platforms like smallest-s2s leveraging the model's capabilities.
Since its launch, Phonon has been downloaded over 200,000 times within the first two weeks and continues to trend on Hugging Face. The Open TTS Leaderboard, a relatively new project by Hugging Face, offers developers a centralized platform to compare open TTS models and listen to them side by side. Evaluating over 9,000 TTS models across 23 languages, the leaderboard assesses intelligibility (word error rate), speed (time-to-first audio), naturalness (UTMOS for English and KOMOS for Korean), and speaker similarity for voice cloning.
Phonon demonstrates remarkable performance, ranking first in intelligibility for German and Spanish and within the top three for all evaluated languages. Phonon's exceptional speed in CPU streaming benchmarks further highlights its advantages over larger models.
Rewritten by urgent.news; may contain errors — check the original.
Source: huggingface.co