Keeping Two Faces From Swapping Mid-Verse: Identity Preservation in AI Music Videos
The problem nobody warns you about You train a model to turn two photos into a short music video. The first three seconds look great. Then, around second five, the guy on the left suddenly has the other guy's jaw. By second eight, their faces have traded places entirely. If you have built any kind of face-to-video pipeline, you know this failure mode well. It has a name: identity drift. And for a…
Identity drift is a common issue when training models to turn two photos into short music videos, where the faces of the performers swap places in the generated video. This problem, known as identity drift, is particularly challenging for videos featuring two people, such as rap duets or covers by friends singing together. The single hardest thing to get right in such a performance video is to maintain the correct identity of the two faces across the entire clip.
This article will explain why identity drift occurs, the engineering decisions that can be made to prevent it, and provide a walkthrough of the failure modes and levers involved.
Single-face face-to-video problems are generally easier to solve since the pipeline can rely on strong priors. The face can be detected, an identity embedding extracted, and every generated frame conditioned on that embedding. However, identity drift becomes significantly more difficult when dealing with two faces. The model must determine which face is which at each frame, even as the performers interact and move their heads.
This attribution problem becomes more complex as the performers trade lines, lean towards the microphone, or overlap in the frame. The motion of one face can bleed into the other, leading to a final product where the two people have converged into a single averaged face, creating the classic "they merged" artifact.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.