I built a local-first AI video dubbing pipeline with WhisperX + VoxCPM2
I wanted to dub short videos without sending the source media to a hosted dubbing platform. The hard part is not producing one synthetic sentence. It is keeping the original speech timing, preserving the background, translating the transcript, cloning a voice, and rebuilding a video that still feels like the same video. That led me to build Video Dubbing Translator , a local-first, browser-based…
A reporter has built a local-first AI video dubbing pipeline using WhisperX and VoxCPM2. The challenge lies not just in producing one synthetic sentence, but in preserving the original speech timing, background, voice, and rebuilding a video that still feels authentic. This led to the creation of Video Dubbing Translator, a browser-based pipeline for short-form video dubbing.
The process involves video input, vocal/background separation, translation, voice cloning, fitting the duration, and reconstruction. The interface is designed to show inspectable stages rather than a single opaque "dub this" button. While the translation currently uses Google Translate, the media processing and speech synthesis run locally, making the pipeline partially offline.
VoxCPM2 was chosen over XTTS-v2 due to its reference-voice cloning, 48 kHz speech output, and an official model with an Apache-2.0 license. An end-to-end test on a 15-second English clip showed that the translated Turkish result matched the original timings, and the final audio recovered all intended Turkish sentences when run through Turkish ASR. The pipeline outputs H.264/AAC video with 48 kHz stereo audio, with explicit audio contract to catch format problems early.
While the current implementation is limited by Google Translate, CPU synthesis speed, and the lack of production-grade speaker diarization, the project is MIT-licensed and open for contributions. The repository includes browser UI, processing stages, setup notes, and demo links. Feedback is sought on voice similarity, timing behavior, installation on different GPUs, multilingual edge cases, and handling multiple speakers.
The goal is to create a reproducible local-first workflow that can be inspected and improved upon, rather than a black-box hosted service.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
