Urgent.News

What's breaking now, across thousands of outlets.

Tech

Tuning vLLM on NVIDIA vGPUs

Introduction Earlier in this series I set up the basic platform: vLLM serving the model, LiteLLM as the gateway and an ingress in front. I kept the vLLM flags quite conservative there on purpose. In this part I want to go through what I actually changed afterwards, and why. My setup: the GPUs are not passed through to the Kubernetes nodes, they are NVIDIA vGPUs on vSphere. Each node VM gets a…

Tuning vLLM on NVIDIA vGPUs presented several challenges, primarily related to memory constraints and performance optimizations. vGPU memory isn't the same as what nvidia-smi reports, as GPUs reserve memory for themselves, leaving less available for vLLM. The solution was to adjust the GPU memory utilization parameter to find a balance between memory usage and headroom. Notably, the --enforce-eager flag, which disables CUDA graphs, negatively impacts performance on vGPUs, so it was avoided.

Another challenge was fitting a 250k token context window into the limited memory, which was achieved by using the fp8 dtype for the KV cache. This halved the memory per token compared to bf16, allowing for more context to be stored. However, the fp8 KV cache is lossy, so quality considerations should be taken into account when deciding on the dtype.

Setting --max-num-batched-tokens to larger values didn't improve performance, as the prefill process was compute-bound, meaning there was no idle capacity left for a bigger batch to utilize. Therefore, the batch size was reverted to 4096 for optimal results. Lastly, the vision encoder and warmup with dummy images were disabled to conserve memory for text-based tasks, which also improved performance.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Check Point Patches a Root Daemon That Accepts Commands From Any Local Process

TL;DR CVE-2026-123039 is a local privilege escalation in Gaia OS, the operating system that runs on Check Point security appliances, where a daemon named dynamic_server executes commands as root.

  • Local privilege escalation vulnerability in Gaia OS, CVE-2026-123039
  • Dynamicserver daemon executes commands as root via world-writable Unix socket
  • Check Point released hotfixes; exploit active in several dozen customer regions

More from Tuesday 17 November →