Running a Jev-Style Decision Model on One TPU v6e: What Fits, What It Costs, and What Changes From a GPU
This article provides a step by step guide to running a Jev-style decision model on one Google Cloud TPU v6e chip with Gemma 4 and vLLM, and compares it with the same read on one NVIDIA L4 GPU. The measurement was pre-registered, and every per-item output is committed. One v6e chip serves Gemma 4 E2B, E4B and 12B at bf16 and a 26B-A4B fp8 build; no 31B checkpoint loads. The same checkpoints give…
This guide explains how to run a Jev-style decision model on a Google Cloud TPU v6e chip using Gemma 4 and vLLM, and compares it with the same task performed on an NVIDIA L4 GPU. The TPU can serve Gemma 4 models in E2B, E4B, and 12B configurations in bf16 precision, while the GPU can handle the same models in bf16 precision. The 26B-A4B and 31B models cannot be loaded onto the TPU due to memory constraints.
The 26B-A4B is quantized to fp8 and available, while the 31B is available in w4a16 and 4-bit formats but also exceeds the TPU's memory limit. The article provides a step-by-step process to pre-register measurements, select appropriate checkpoints, launch the TPU, serve models using vLLM, and retrieve label probabilities. The results show that the TPU is faster than the GPU, with the 12B model decision taking 27-33 ms on the TPU compared to 61 ms on the L4.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
