Ming-3.0-tts on a 16 GB card: the GPU sat mostly idle, and why a bigger one barely helps
When a speech model runs slower than real time, the reflex is to rent a bigger GPU. With Ming-3.0-tts-0.5B that buys very little. An RTX 2000 Ada sat idle more than two thirds of the time. An RTX 4090 worked at most one moment in five, and the speedup was about 17%. The model comes from the inclusionAI team at Ant Group. They released weights and code under MIT, with their own benchmarks and a…
When a speech synthesis model operates slower than real time, it is common to seek a more powerful GPU. However, this does not bring significant improvements for the Ming-3.0-tts-0.5B model. An RTX 2000 Ada GPU spent more than two thirds of the time idle, while an RTX 4090 was only able to work at most once in five instances, resulting in a speedup of about 17%.
The model was provided by the inclusionAI team at Ant Group, who released the weights and code under an MIT license, along with their benchmarks and a list of limitations. The model can copy a voice from a few seconds of sample audio, generate one from a text prompt describing it, and convert inline tags like [Laughter] or [Sigh] into sounds.
The model's documentation only mentions H800-80GB and H20-96G cards, without specifying the required memory. We ran the unmodified code on a small 16 GB card to replicate the results. One of the main factors affecting the model's performance is the decode loop, which constructs the audio step by step, 80 milliseconds per step. Between steps, the CPU prepares and launches the next step.
The GPU's fast bus is held back by a slow ticket check at every stop. Before deciding on hardware for a text-to-speech (TTS) model, it is crucial to measure the GPU utilisation using nvidia-smi. If the GPU is idle for most of the time, a faster card will not provide a significant speedup. In our tests, no card with less than 16 GB of memory performed well.
We tried using two 12 GB RTX 3080 Ti machines and an 8 GB card, but neither worked. To work around this limitation, we simulated a 16 GB card using PyTorch's torch.cuda.set_per_process_memory_fraction parameter to limit the GPU memory the process could utilize. With memory caps of 8, 4, and 3 GiB, the output WAVs had the same md5 checksum as the uncapped run, and the speed did not change.
At a cap of 2.75 GiB, the setup failed with a CUDA out of memory error. To avoid this issue, we estimated that a 4 GB card would be required, accounting for roughly 0.4 GB for the driver.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
