How to Run an AI Model on Almost Any Hardware: A Practical Guide to Tiny Local LLMs

This month, a developer successfully ran a language model on a 2007 Sony PSP handheld device. The device has a 333 MHz MIPS processor, 64 MB of RAM, and streams English text at 1-2 tokens per second. The model used is Karpathy's stories15M, int8-quantized down to 17 MB, and runs in around 1,100 lines of pure C code. One token completion takes about a minute to complete.

The author highlights that people often pick a model first, then discover it doesn't fit their hardware. This PSP experiment is an example of running the right hardware-first approach, starting with the available hardware and finding the largest model that fits it with some headroom. The article provides a practical guide to running tiny local LLMs on various hardware.

The trend of smaller models is growing, with notable examples like the PSP build and the Falcon-H1-Tiny 90M family. Quantization has improved for smaller models, with Unsloth Dynamic 3.0 GGUFs achieving up to 10% better top-1 accuracy at the same file size compared to other providers. The author emphasizes that tiny models are not suitable for general chat or agents, but are great for routing and classification, autocomplete, and structured extraction.

To size a model to your hardware in 15 minutes, the author suggests measuring real free memory, budgeting more than the file size, and picking a quant tier based on available memory. For example, UD-Q4_K_XL is suitable for 16 GB machines, UD-Q2_K_XL for 9.83 GB, and UD-IQ1_S for 6.2 GB. The goal is to cover everyday local use cases like chat, tool calling, and light coding without exceeding 500 MB of quantized memory.