Building a 512-LED wall that dances to my music: FFT, ESP32, and a custom serial protocol

The project involved constructing a 512-LED wall that responded to music. The design comprised eight 8×8 panels arranged as a 32×16 grid, controlled by an ESP32. The analysis of the audio stream was performed on a PC, while the ESP32 served as a display unit, receiving frames over USB serial and handling LED updates. The PC decoded the mp3, calculated the Fast Fourier Transform (FFT) into 32 frequency bands, and created an RGB frame for the LEDs.

This frame was then serialized and sent to the ESP32. The architecture was engineered to ensure zero duplicated state between the PC and ESP32, placing the analysis on the PC and the display on the ESP32. The protocol between the two components was carefully designed, with specific frame structures and limits to prevent corruption.

The ESP32 received 1542-byte frames, applied a current limiter, and communicated with the PC using a custom serial protocol. The first version of the system used a fire-and-forget approach, but it resulted in corruption due to buffer overflow. The solution was to implement lock-step flow control, where the ESP32 sent an ACK byte after every frame, allowing the PC to hold the next frame until the ACK was received.

This ensured zero frame loss during LED updates and maintained audio-synced visuals. The project was written in Python, using libraries such as soundfile, numpy, and FastLED, with a focus on vectorized operations and careful protocol design.