Edge0 streams MoE experts off SSD to fit 35B in 3 GB
Where does a 35B model go when it only takes 2.9 GB of RAM? I went into Edge0 to find out, and I came out with a different mental model of what a local model costs. What shipped Edge0-AI open-sourced Edge0 the other day: a streaming inference engine under Apache 2.0, plus two preview models built on open sparse MoE bases. Edge0-35B-A3B sits on Qwen3.5-MoE 35B-A3B, 40 layers and 256 experts.…
Edge0 has released open-sourced streaming inference engine Edge0-AI, showcasing an impressive 35B mixture-of-experts model that demands only 2.9 GB of RAM. The checkpoint, compressed at 4-bit precision, occupies 19.6 GB of space on disk, with an additional 2 GB allocated for higher-precision elements such as embeddings and attention weights.
Despite its size, the model only actively uses 2.9 GB of memory, with the majority of its parameters being memory-mapped and only the relevant experts loaded during token processing. This approach maximizes efficiency by keeping peak memory consumption proportional to the active set rather than the total parameter count. The project's benchmark shows the 35B model achieving a decoding speed of 14.9-17.7 tokens per second on a Mac mini M4 Pro with 24 GB of unified memory, while the 8B model achieves 23.9-25.3 tok/s.
The runtime's aggressive routing strategy allows for efficient expert selection, with only 4 out of the 256 experts being called per token. The model's compact footprint makes it well-suited for users with limited RAM on Apple Silicon Macs, although a potential concern arises from the reliance on the operating system's file cache for expert reads, which could strain drives over prolonged use.
Despite being a preview model, Edge0's architecture offers a promising glimpse into the future of resource-efficient local model deployment.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
