I Built a Local AI Music Studio

A local music generation and remix studio called Miso was developed by its creator. The project began as a simple idea to provide an interface for experimenting with local music models. However, it quickly evolved into a more complex system. Miso now includes a database, Docker Compose stack, job queue, model manager, audio editor, waveform player, and multiple GPU models that consume 40 GB of disk space.

The primary goal of Miso is to allow users to experiment with music generation and remixing, including editing specific sections of a song, separating stems, and applying different styles to the same melody. The studio supports several music and audio model families, such as ACE-Step 1.5, YuE2, MiniMax Music 3, HeartMuLa, and Stable Audio 3.

It also includes features like repainting, stem separation, voice conversion, and audio-to-MIDI transcription. Users can load a take, drag across a section of its waveform, and replace only that part, which is particularly useful when only a small portion of a song needs modification. Miso also allows users to split a track into separate stems, including vocals, drums, bass, and other elements.

These stems can then be mixed, soloed, muted, or altered using voice conversion models. The latest release, 1.4.0, connects MIDI transcription to YuE2's score support. This allows users to separate a song, transcribe the vocal stem, convert it into an ABC melody, and feed the score into YuE2 with a different style. This enables users to take a known melody, such as Twinkle Twinkle Little Star, and use it as the basis for a new arrangement.