I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read
8GB of VRAM isn't the bottleneck, your settings probably are
The VRAM capacity of a GPU often dictates the size of local AI models that can be self-hosted. However, there are ways to optimize smaller systems with limited VRAM to run large models. One such method involves using the llama.cpp runtime and a flag called --n-cpu-moe N. This flag helps optimize a system with lower VRAM by moving dormant expert layers to system RAM while keeping the active parts of the network on the GPU.
By testing this approach on an 8GB RTX 3070 Mobile GPU, it was discovered that it could run a 35B model, Qwen3.6-35B-A3B, at a speed of 33 tokens per second, which is significantly faster than the user's previous expectations. This optimization enables users to access and work with larger AI models on the go, even in places with no network connectivity.
Written by urgent.news from XDA Developers's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
