I turn TikTok trends into one-photo AI video templates. Here's what keeps breaking

TikTok trends are popular, but most people don't want to write prompts. They prefer to upload their own face or friends' faces into existing clips. This is the concept behind AIMakeVideo, where the site takes a popular TikTok trend, rebuilds it as a template, and allows users to upload a photo to get the trend back as their own.

The process involves breaking down reference clips, generating a depth video, creating a plate without people, and then feeding the depth video, plate, face crop, full photo of each person, and a prompt into a model. The model then places the people into the trend with the original audio track, and charges a flat price.

However, there were numerous issues during the pipeline. For example, the reference clip often leaked into the output. In one case, a cross necklace appeared on a dancer not present in the reference clip, due to the depth map carrying shape information. To fix this, the prompt was updated to forbid necklaces, hats, caps, and chains unless they were visible in the uploaded photo.

Another issue was the model adding black mouths to singing trends, as depth maps lack mouth information. This was resolved by using a softer, feathered grey hole in the prompt instead of a black patch.

Additionally, users were uploading dog photos to dance templates, even though the pipeline was designed for replacing people. A face check was added to the browser before upload, distinguishing between real people and pets or characters. Users could also choose their preferred pipeline, with real people charged full price and pets or characters charged a different rate.

Another challenge was creating one-photo templates for two people. The model kept the uploaded photo's waist-up crop, resulting in a hard edge appearance with no camera movement. The solution was to cut the photo down the middle and send each half as its own person. The rule was updated to one person per reference image.

Lastly, the Tiny Desk template had five parts, but a single photo was required. To address this, the lead singer and keyboard player were never on screen simultaneously. If only one photo was uploaded, the user appeared in every shot. This policy was kept as it catered to users who wanted to see themselves in the video.