{
  "id": 18998242,
  "title": "Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050",
  "url": "https://urgent.news/2026/11/08/six-serving-engines-one-8-gb-gpu-gemma-4-e2b-on-an-rtx-3050",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-08T19:08:37.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/gde/six-serving-engines-one-8-gb-gpu-gemma-4-e2b-on-an-rtx-3050-ad7"
  },
  "original_language": "en",
  "account": "This article provides a detailed comparison of six inference serving engines that run the Gemma 4 E2B model on an 8 GB RTX 3050 GPU. The six engines are llama.cpp, vLLM, SGLang, mistral.rs, PyTorch, and LiteRT-LM. The goal is to determine the best performing engine for running the model on a small local GPU.\n\nEach engine is configured with the same Gemma 4 E2B weights, prompts, and load sweeps. The model is tested with 1,319 GSM8K and 400 BFCL questions to measure answer quality with and without the Google's E2B drafter. The results show that the answer quality doesn't vary much between the engines, with GSM8K scores ranging from 0.8992 to 0.9030 and BFCL scores from 0.9125 to 0.925. The drafter further boosts performance, with vLLM, llama.cpp, SGLang, and LiteRT-LM having improved scores compared to mistral.rs.\n\nWhen it comes to throughput, vLLM leads with 1,651 tokens per second, while SGLang follows with 1,711. In contrast, llama.cpp and mistral.rs achieve 633 and 308 tokens per second, respectively. When running at their best settings, vLLM remains the fastest for 1 to 16 requests, and SGLang leads vLLM by 1 to 3% for 32 and 64 requests on long replies.\n\nThe article also highlights the importance of properly sizing each engine for the 8 GB GPU. vLLM uses 85% of the GPU memory upfront, while SGLang reserves 80% and captures CUDA graphs for 64 running requests. llama.cpp and mistral.rs have specific settings for running 64 slots and 4 requests with a fixed cache size.\n\nTo conduct the tests, each engine is set up in its own directory with a configuration file, a serving script or Makefile, and measurement files in a benchmarks/runs directory. Only one engine serves at a time, and none of the engines fit on the 8 GB GPU simultaneously. The setup requires an NVIDIA RTX 3050 GPU, CUDA toolkit, Python 3.14 interpreter, pip, and Rust toolchains for mistral.rs. The Gemma 4 E2B model weights and checkpoints used in the tests are specified for each engine.",
  "summary": "This article provides a step by step comparison of six inference serving engines on one consumer GPU: llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and Google's LiteRT-LM, each serving Gemma 4 E2B on Google's QAT 4-bit weights from an 8 GB RTX 3050 in a desktop. https://github.com/xbill9/gemma4-dev The question is which engine to run when the model lives on a small local card that…",
  "key_points": [
    "vLLM leads in throughput with 1,651 tokens per second",
    "SGLang follows with 1,711 tokens per second",
    "llama.cpp and mistral.rs achieve lower throughput at 633 and 308 tokens per second"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}