g5g vs g6 for LLM Serving: the Same Code, and 3.7x the Throughput
This article compares two AWS GPU instance families for serving a small language model, using a payload that is byte-identical on both. The older family loses 87% of decode to dtype conversion, and nothing in any log, metric or health check says so. The code is here: https://github.com/xbill9/gemma4-dev The Two Instances g5g.2xlarge pairs a Graviton2 (aarch64) host with an NVIDIA T4G — Turing, SM…
This article compares two AWS GPU instance families, g5g.2xlarge and g6.2xlarge, for serving a small language model. Both instances have identical payloads, but the g6.2xlarge shows three and a half times the throughput of the g5g.2xlarge. The g5g uses NVIDIA T4G with Turing architecture, while the g6 uses NVIDIA L4 with Ada architecture.
The study used a hand-written pure-JAX port of the Google Gemma-4-E2B model, with byte-identical inputs and outputs. The g5g loses 87% of decode to dtype conversion, while the g6 bypasses this issue entirely. The main bottleneck on the g5g is dtype conversion and an fp32 gemvx path, which the g6 avoids by running at full memory-bandwidth roofline.
The source material provides details on the hardware, setup, and benchmark results, but does not speculate on the implications or future directions.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
