Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
Article URL: https://github.com/leonickson1/Swiftlet Comments URL: https://news.ycombinator.com/item?id=49158333 Points: 186 # Comments: 80
Swiftlet is a Swift and Metal runtime that enables running large Qwen Mixture-of-Experts models locally on Apple devices. By streaming expert weights from storage, it allows 35B and 80B models to operate with minimal RAM, including on iPhones. These models generate correct, validated output and run with low RAM requirements, such as 2.5 GB for the 35B on an iPhone 17.
The models activate only about 3B of their parameters per token, allowing them to chat and write like large models while recalling facts more like smaller ones. Swiftlet achieves this by routing every token to a subset of the experts, with no growing KV cache for any context length. The Metal kernels are tested against CPU references, and fast GPU kernels are verified for identical outputs.
The open-source project requires Apple Silicon, macOS 14+ or iOS 17+, and free SSD space for the model container.
Written by urgent.news from Hacker News Best's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
