Strata: Running a 125-Billion-Parameter Model on Your Own Gaming PC

Running a 125-billion-parameter model on a single 12-GB consumer GPU has never been more feasible thanks to Strata, an open-source project with 17,002 stars on GitHub. Developed by MIT and released in C++, Strata achieves this feat through low-bit quantization (Q2_0 / IQ2_XS) applied to the Qwen3.8-Flash-Next model and a purpose-built inference engine.

This compression allows a model typically requiring a server to run on a gaming PC, as demonstrated on two ordinary gaming PCs: an RTX 5070 (12 GB) paired with a Ryzen 5 7600 using Q2_0, and an RX 9070 XT (16 GB) with a Ryzen 9 3900X using IQ2_XS. The authors measured the performance, with RTX 5070 using Q2_0 achieving 94 tokens per second, while the RX 9070 XT using IQ2_XS reached 79 tokens per second.

For context, human reading speed is approximately 5-10 tokens per second, so a $1,000 gaming rig can now output a 125B model's responses faster than one can read them. Strata's value proposition lies in moving self-hosting from data centers to desktops, enabling private deployments with a single machine containing a 12 GB card. The project offers an OpenAI/Anthropic-compatible API on localhost, allowing existing clients and agent frameworks to connect by simply changing the base_url.

Strata provides Windows and Linux installers, supported by both NVIDIA and AMD. While quantization precision comes at a cost, with potential limitations in complex reasoning and long-horizon tasks, Strata is designed for use by individuals or small teams with privacy concerns or budget constraints, rather than high-precision production inference.