{
  "id": 20407297,
  "title": "Tuning vLLM on NVIDIA vGPUs",
  "url": "https://urgent.news/2026/11/17/tuning-vllm-on-nvidia-vgpus",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-17T09:00:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/florianlutz/tuning-vllm-on-nvidia-vgpus-5bo3"
  },
  "original_language": "en",
  "account": "Tuning vLLM on NVIDIA vGPUs presented several challenges, primarily related to memory constraints and performance optimizations. vGPU memory isn't the same as what nvidia-smi reports, as GPUs reserve memory for themselves, leaving less available for vLLM. The solution was to adjust the GPU memory utilization parameter to find a balance between memory usage and headroom. Notably, the --enforce-eager flag, which disables CUDA graphs, negatively impacts performance on vGPUs, so it was avoided.\n\nAnother challenge was fitting a 250k token context window into the limited memory, which was achieved by using the fp8 dtype for the KV cache. This halved the memory per token compared to bf16, allowing for more context to be stored. However, the fp8 KV cache is lossy, so quality considerations should be taken into account when deciding on the dtype.\n\nSetting --max-num-batched-tokens to larger values didn't improve performance, as the prefill process was compute-bound, meaning there was no idle capacity left for a bigger batch to utilize. Therefore, the batch size was reverted to 4096 for optimal results. Lastly, the vision encoder and warmup with dummy images were disabled to conserve memory for text-based tasks, which also improved performance.",
  "summary": "Introduction Earlier in this series I set up the basic platform: vLLM serving the model, LiteLLM as the gateway and an ingress in front. I kept the vLLM flags quite conservative there on purpose. In this part I want to go through what I actually changed afterwards, and why. My setup: the GPUs are not passed through to the Kubernetes nodes, they are NVIDIA vGPUs on vSphere. Each node VM gets a…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}