Google's QAT Gemma 4 26B-A4B on One TPU v6e: 15.6x the KV Cache and 1.9x the Throughput of FP8
This article provides a step by step guide to serving Google's quantization-aware-trained (QAT) Gemma 4 26B-A4B on one Google Cloud TPU v6e chip with vLLM, and compares it with the FP8 build that is the only 26B serving on one chip today. Every per-record output, log and script is committed. The QAT 26B serves on one v6e chip at 17.43 GiB of HBM, with 53,888 tokens of KV cache and 1,283 output…
Google's QAT Gemma 4 26B-A4B model, trained with 26 billion parameters and 128 experts per layer, requires 48.07 GiB of GPU memory when using bf16 precision. Google Cloud's TPU v6e chip provides 28.74 GiB of usable HBM, which is insufficient for serving the FP16 model directly. In contrast, RedHat's FP8 build achieves 668 output tokens per second but consumes 27.99 GiB of memory.
A recent comparison shows that Google's QAT version and RedHat's FP8 build perform similarly in a 3,880-record classification suite. The article details Google's process of converting their QAT 26B-A4B model to a 4-bit (W4A16) format, which can be served on a single TPU v6e chip. The process involves recovering the quantization step used during training, repacking the weights and attention scores into compressed tensors, and verifying the accuracy of the modified checkpoint.
The final W4A16 checkpoint is available on Hugging Face under the name xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
