{
  "id": 11482694,
  "title": "Gemma 4 QAT on One TPU v5e: What Runs and What Doesn't",
  "url": "https://urgent.news/2026/10/02/gemma-4-qat-on-one-tpu-v5e-what-runs-and-what-doesnt",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-02T16:59:51.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/gde/gemma-4-qat-on-one-tpu-v5e-what-runs-and-what-doesnt-2gii"
  },
  "original_language": "en",
  "account": "This article offers a step-by-step guide on repacking Google's quantization-aware-trained (QAT) Gemma 4 weights for serving on a single Google Cloud TPU v5e chip, testing their performance across various metrics. The QAT builds allow the Gemma 4 model, ranging in size from E2B to 26B parameters, to run on one v5e chip without issue. The 4-bit repack stores the QAT grid values using int4 weights and activations, while the 8-bit repack employs int8 weights and activations, which show a slight speed advantage. Despite the 4-bit repack's slightly lower performance compared to the 8-bit repack, both models match the performance of Google's own 4-bit exports at the same speed. The 12B model is the largest Gemma 4 size that can fit on the TPU v5e chip, with a weight size of 11.31 GiB and 675 output tokens per second at 16 requests. Overall, the repacking process enables Gemma 4 models to run efficiently on a single TPU v5e chip while maintaining competitive performance against Google's own 4-bit exports.",
  "summary": "This article provides a step by step guide to repacking Google's quantization-aware-trained (QAT) Gemma 4 weights for vLLM and serving them on one Google Cloud TPU v5e chip, with every build scored for classification, math, tool calling, throughput and long prompts. Every per-record output, log and script is committed. On one v5e chip the repacked QAT builds serve every Gemma 4 size from E2B to…",
  "key_points": [
    "Gemma 4 models run efficiently on a single TPU v5e chip",
    "4-bit repack stores QAT grid values using int4 weights and activations",
    "8-bit repack shows slight speed advantage over 4-bit repack"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}