A Dense 31B on an 8 GB GPU: Gemma 4 31B on an RTX 3050
This article provides a step by step guide to serving Gemma 4 31B, Google's quantization-aware trained (QAT) dense model, on an 8 GB NVIDIA RTX 3050 with llama.cpp. The model file is 17.7 GB, so it cannot fit on the card. llama.cpp puts as many layers as will fit on the GPU and runs the rest on the CPU. The measures are how fast it generates, how much context fits, what the drafter adds, and…
This article provides a detailed guide on how to serve Gemma 4 31B, Google's quantization-aware trained (QAT) dense model, on an 8 GB NVIDIA RTX 3050 GPU using llama.cpp. The model file is 17.7 GB and cannot fit entirely on the GPU, but llama.cpp manages to run as many layers as possible on the GPU and the rest on the CPU. The article also discusses the performance of the 31B model compared to its 12B and 26B siblings, with the 31B being the most accurate but also the slowest.
The measures include generation speed, context length, drafter additions, and whether the answers surpass previous models. The article is the sixth part of a series, with the previous parts covering Gemma 4's MTP Drafter, E2B and E4B measurements, the context size for a 12B on an 8 GB GPU, and the 26B mixture-of-experts running on an 8 GB GPU.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
