Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

This article provides a detailed comparison of six inference serving engines that run the Gemma 4 E2B model on an 8 GB RTX 3050 GPU. The six engines are llama.cpp, vLLM, SGLang, mistral.rs, PyTorch, and LiteRT-LM. The goal is to determine the best performing engine for running the model on a small local GPU.

Each engine is configured with the same Gemma 4 E2B weights, prompts, and load sweeps. The model is tested with 1,319 GSM8K and 400 BFCL questions to measure answer quality with and without the Google's E2B drafter. The results show that the answer quality doesn't vary much between the engines, with GSM8K scores ranging from 0.8992 to 0.9030 and BFCL scores from 0.9125 to 0.925. The drafter further boosts performance, with vLLM, llama.cpp, SGLang, and LiteRT-LM having improved scores compared to mistral.rs.

When it comes to throughput, vLLM leads with 1,651 tokens per second, while SGLang follows with 1,711. In contrast, llama.cpp and mistral.rs achieve 633 and 308 tokens per second, respectively. When running at their best settings, vLLM remains the fastest for 1 to 16 requests, and SGLang leads vLLM by 1 to 3% for 32 and 64 requests on long replies.

The article also highlights the importance of properly sizing each engine for the 8 GB GPU. vLLM uses 85% of the GPU memory upfront, while SGLang reserves 80% and captures CUDA graphs for 64 running requests. llama.cpp and mistral.rs have specific settings for running 64 slots and 4 requests with a fixed cache size.

To conduct the tests, each engine is set up in its own directory with a configuration file, a serving script or Makefile, and measurement files in a benchmarks/runs directory. Only one engine serves at a time, and none of the engines fit on the 8 GB GPU simultaneously. The setup requires an NVIDIA RTX 3050 GPU, CUDA toolkit, Python 3.14 interpreter, pip, and Rust toolchains for mistral.rs. The Gemma 4 E2B model weights and checkpoints used in the tests are specified for each engine.