How I Turned QR-Coded Chips Into a Multi-Track Musical Instrument on iOS Without Core ML

Inside a transparent window on a flat board, a phone rests face-up. A plastic chip, containing a QR code, is slid underneath the window, triggering the start of a drum loop. Another chip is added, bringing in a bass line. Removing one chip causes the drum track to fade away, proving the concept that physical objects on a table become tracks in a mixer.

The app uses QR codes and a phone's camera to control a 24-channel audio mixer, with each chip containing a two-digit number mapping to a specific track. No buttons, no touchscreen, no MIDI controller are required. The entire recognition pipeline is executed using a single AVFoundation class that has been part of iOS for years. The project began in September 2021, with the first version featuring camera, QR scanning, audio engine, and fade functionality.

After a year and a half, voice control, sample persistence, per-track volume control, and packaging were added, turning it into an early-stage venture in music education. The app has been refined over time, with the camera switched from rear to front-facing and the technical decisions shared as an MVP. Initially, the app used Core ML for chip recognition, but QR codes were later adopted for their lighting-independence and ease of implementation.