Dev proves LLMs will run on anything – even a $10 microcontroller

A developer named SlvDev has managed to get a small language model running on an ESP32 microcontroller, which costs less than $10. This might seem impossible, as these microcontrollers are typically used for tasks like remote sensors and IoT applications, not generative AI models. However, SlvDev managed to achieve this by using techniques such as quantization and per-layer-embedding (PLE) to shrink the model's footprint.

The original model was 28.9 million parameters, but SlvDev managed to compress it to 10,000 times smaller, TinyStories with 28.9 million parameters. Even this smaller model requires significant memory, but by offloading some of the weights to flash storage, SlvDev was able to reduce the memory needed to just 2 MB. The output head, embeddings, and KV cache are kept in the microcontroller's PSRAM, while activations are handled in the 520 KB of SRAM.

This results in the microcontroller being able to generate 9.88 tokens per second, faster than the average person can read. While this achievement is impressive, the TinyStories model is not capable of much beyond generating short, coherent stories. However, it's still a proof-of-concept and demonstrates that even small microcontrollers like the ESP32 can run language models.