How I built an AI rap video generator as a solo maker
A few weeks ago my feed was full of the "Hotel Lobby" rap edits: two people in an orange booth, one hanging mic, rapping a famous song with swapped-in celebrity faces. I wanted a version where you use your own photo and get an original rap instead. So I built one. Here's how the pipeline works and what I learned. What it does You upload one photo (or two for a duo), pick a scene and a beat, and…
A few weeks ago, I encountered a trend on social media featuring "Hotel Lobby" rap edits, where two people in an orange booth, each with a hanging mic, rapped a famous song while wearing swapped-in celebrity faces. Inspired by this, I set out to create my own AI rap video generator. Here's how the process works and what I discovered along the way.
This generator allows users to upload a single photo (or two for a duo), choose a scene and beat, and type in a topic like "my golden retriever who steals socks." In just a few minutes, users receive a 5-15 second MP4 where the face raps original lyrics synced to a new beat.
The AI generation pipeline is broken down into distinct steps to ensure each component can fail and retry independently. First, moderation checks the topic and any custom lyrics before any expensive processing occurs. Next, an LLM (Anthropic's Claude) writes bars of lyrics based on the topic, with the prompt adjusting the number of lines to match the video length. Users can also input their own lines, one per bar.
The generated lyrics, beat style (such as Trap, Boom Bap, Drill, or Old School), and chosen scene are then input into ByteDance's Seedance 2.0 family, which creates the music, voice, and lip-synced video in a single step. Duo mode enables users to upload two photos, resulting in two performers sharing the same frame who trade lines during the video. Ensuring both faces remained in sync while avoiding the creation of additional people proved to be a significant challenge.
After the video is rendered, it's stored in object storage, and the user can download an MP4 file in 9:16, 1:1, or 16:9 aspect ratios. Rendering takes between 2-4 minutes, so the frontend polls a task record instead of keeping a request open indefinitely. Credits are deducted when a task starts and refunded automatically if the render fails, preventing users from having to contact the creator to retrieve credits for a broken video.
To manage costs, the service uses credits instead of a flat subscription model. Credits are deducted upon task initiation and refunded automatically if the rendering process fails, ensuring users aren't charged for unsuccessful videos. The stack consists of TanStack Start (React 19, Vite) hosted on Cloudflare Workers, along with Drizzle ORM and better-auth for user authentication.
I learned that releasing a free tool early can help attract users before they commit to paying for videos. Testing with pets early in the development process is also beneficial, as people often enjoy making their dogs rap. Furthermore, focusing on a smaller number of well-tuned scenes rather than a larger quantity of generic ones can lead to better results.
Finally, I encourage feedback on the pipeline, particularly from those who have encountered challenges with lip-sync on non-human faces. The generator is currently available at rapvideo-ai.io, where new accounts receive free credits for their first short video.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.