Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4

Amazon SageMaker's real-time endpoints provide a quick way to deploy models for inference. In this article, the author compares the performance of two Gemma 4 models, one in full-size bf16 format and the other with quantization-aware trained (QAT) 4-bit weights, both hosted on an NVIDIA L4 instance in the US East (us-east-2) region.

Using the vLLM framework and AWS tools, the author found that the QAT checkpoint decodes text at 105.1 tokens per second, a significant 2.05 times faster than the bf16 checkpoint which decodes at 51.3 tokens per second. The QAT model also served 1077.25 tokens per second with 16 parallel requests, compared to 619.1 tokens per second for the bf16 model.

Both models achieved the same accuracy on a set of 40 checked questions. The deployment process involves creating four objects: Model, Endpoint config, Endpoint, and a running HTTPS service. SageMaker manages the container, checks for health, routes traffic, and logs to CloudWatch.

The comparison was conducted by running the same three measurements, decode time, parallel requests, and answer accuracy, against each endpoint. The QAT checkpoint saves 18% GPU memory by using 4-bit weights instead of bf16. This memory saving is attributed to the smaller size of the transformer body, which is the only part of the model compressed in 4-bit format. The QAT checkpoint's header shows this memory distribution across different parts of the model.