Amazon's Prime Video is using AI to make dubbed actors' mouths match the dialogue

Amazon Prime Video is leveraging artificial intelligence to enhance the quality of dubbed English versions of foreign-language shows. The technology targets the synchronization of actors' lip movements with the translated dialogue. Initially applied to the German-language series Maxton Hall, the feature is set to roll out to additional titles in the future, though specifics on future applications have not been disclosed.

Maxton Hall, Amazon's most-watched international original series, is being used as a test case for the technology. The AI system works in tandem with human-recorded dubs, adjusting the lip movements after translating and recording the dialogue. This approach aims to create a more natural viewing experience by aligning the English audio with the visual cues on screen.

Prime Video's vice president of technology, Raf Soltanovich, emphasized the potential of this technology to make content more immersive for viewers and to expand the reach of creators' stories globally. The feature currently operates under creative oversight, ensuring the integrity of the artistic vision is maintained. Despite the growing interest in AI translation and dubbing tools from competitors like Meta and YouTube, Amazon's method stands out by maintaining human involvement in the dubbing process, rather than relying solely on automated voices.