Show HN: I trained a 125M model to autocomplete piano on-device

A researcher trained a 125-million parameter transformer to autocomplete piano performances on-device, achieving real-time results (~108 notes per second on an iPhone 15). The key to success lay in the right MIDI representation, aggressive data cleaning, and applying DPO post-training. The project began over a year ago with the goal of using AI to autocomplete piano music, similar to GitHub Copilot but for piano.

MIDI files represent music as a sequence of events, including key presses, releases, sustain pedal changes, and instrument switches. The author primarily focused on piano-like material, removing or reducing other events. They transformed MIDI events into a discrete sequence suitable for the transformer model, choosing a representation that represented time shifts using delta_onset, chords as multiple notes with zero delta_onset, and avoiding note-off drift by ensuring note duration was explicit.

The model was built around five categorical fields for each note, including pitch, delta, duration, sustain, and velocity, with separate output heads for each field. The transformer backbone processed the notes once, rather than once per field, improving efficiency. The model also incorporated sustain duration preprocessing, extending note duration when the sustain pedal was down.

The author searched for MIDI datasets, mostly focusing on older classical music in the public domain. Cleaning and selecting the data proved more important than simply increasing dataset size. The training objective was cross-entropy over the five output heads, enabling the tracking of accuracy for pitch, duration, and velocity separately. However, the cross-entropy loss had limitations, as music continuation has no single correct answer. Augmenting the data was crucial due to the imperfect nature of live input MIDI files.