The next age of LLMs? Dev gets a small LLM running at 10 tokens a second locally on a $10 microcontroller

A developer named slvDev has successfully run a 28.9-million-parameter language model locally at 9.88 tokens per second on a $10 ESP32-S3 microcontroller, according to recent wire material. The project, esp32-ai, was made available on GitHub under an MIT license in late July 2026 and has since gained over 3,600 stars and more than 470 forks of the underlying code.

To make the model run on such a resource-constrained device, slvDev had to scale down the model due to the limitations of the hardware. Despite this, the model still managed to generate text similar to TinyStories at this speed. The process involved two main steps: quantization and offloading.

Quantization reduced the weights of the model from 16-bit precision to four-bit precision, significantly reducing the memory required. This brought the RAM requirement from nearly 60MB down to 14.9MB. However, the model still needed a large amount of memory to store its parameters, especially the embedding tables that are read rather than computed.

To further optimize memory usage, slvDev offloaded most of the model's parameters into the microcontroller's 16MB flash memory. This reduced the memory required for working memory to around 2MB. The remaining parameters are stored in the flash memory and accessed sparingly, which does not become a bottleneck.

The resulting performance of 9.88 tokens per second is faster than most people read. However, the model's capabilities are limited. It can generate short, mostly coherent stories but cannot answer questions, follow instructions, write code, or know facts. This is due to the small part of the model that performs the reasoning. The developer acknowledges that this approach does not make small models smarter but allows them to run on devices they previously couldn't. The second model, Barista, is designed to answer espresso-related questions.