Microsoft Patents an AI That Can Literally Put Words in People’s Mouths in Real Time

Microsoft has recently filed a patent for an AI-driven technology that could fundamentally alter the way we perceive live video presentations. According to the patent, this innovative system would use AI agents to identify and modify specific words, as well as a speaker's facial expressions and mouth movements, all in real-time. By relying on set parameters or user profiles, the system would seamlessly edit the video and audio just before and after the modified or replaced words, creating a smooth transition.

In a world increasingly dominated by AI-generated deepfakes, disinformation, and misleading information, this patent could represent a new era of filtering tools that subtly manipulate reality on a highly personalized level. Viewers could consume live events without having to listen to content they find unpleasant or disagreeable.

The patent outlines several potential use cases, ranging from real-time translation, erasing filler words like "um" or "ah," and removing speech impediments, to eliminating personal words or colloquial terms.

However, the patent also raises concerns about how this technology might impact our engagement with live footage. Currently, if viewers don't like what they hear, they can simply turn it off. But the Microsoft patent suggests an environment where people could adjust someone else's speech to their personal preferences. This could potentially deepen the formation of personalized bubbles, where individuals are exposed to only the information and perspectives that align with their personal tastes.

Moreover, the patent mentions that the original live video and audio may be recorded and stored for "ground truth," indicating potential misuse and confusion that could arise from such a tool. Live footage that could be transformed by this system includes lectures, presentations, television productions, awards ceremonies, game and cooking shows, and live sports events, among others.

While dubbing and video editing have long existed, the AI-driven system described in this patent could change how audiences interact with live footage. The patent does not directly address the potential implications of using a tool that eliminates errors or erases words and phrases associated with a speaker's local dialect. However, it does acknowledge the possibility of such outcomes, prompting questions about the broader societal implications of this technology.