Two vLLM servers on one GPU: the startup check that bites, and the fix
Someone asked on the vLLM tracker how to run a small embedding model next to a chat model on one GPU, so the card isn't half idle. I tested it on a rented RTX 3090 (24 GB) with vLLM 0.27.1: BAAI/bge-small-en-v1.5 for embeddings and Qwen2.5-1.5B-Instruct for chat, each in its own vllm serve . It works Embedding server at --gpu-memory-utilization 0.10 , chat server at 0.70 . Both came up and served…
A vLLM user inquired about running a small embedding model alongside a chat model on a single GPU, to maximize GPU utilization. Tests were conducted on an RTX 3090 with 24 GB of memory, utilizing vLLM version 0.27.1. The embedding server was configured at 10% GPU memory utilization, while the chat server was set to 70%. Both servers were successfully started, consuming a total of 18,317 MiB of the available 24,576 MiB.
However, starting the chat server first resulted in an error when attempting to start the embedding server. The error message stated that the free memory on device cuda:0 was less than the desired GPU memory utilization for the embedding server. This was because vLLM checks that there is enough free memory on the device for the desired utilization of each model at startup. Starting with a larger memory utilization for the chat model consumed most of the available memory, leaving insufficient space for the embedding model.
The solution was to allocate a smaller memory utilization for the embedding model. Due to the lack of a key-value (KV) cache in the embedding model, it remained relatively small, using between 450 and 580 MiB regardless of the memory utilization percentage set. Setting the embedding model's memory utilization to 3% (0.03) allowed it to start alongside the 90% chat server, using a combined total of 23,110 MiB and 552 MiB on the single GPU.
The performance impact of running both models simultaneously was measured. The embedding server's throughput dropped by 8.6% compared to running alone, while the chat server's output tokens per second decreased by 11.8%. The models ran concurrently for 30 seconds each, with no errors reported during the tests. Running multiple small models on one GPU comes at the cost of reduced performance for each individual model.
It is recommended to leave some headroom, such as using a larger memory utilization percentage for the more resource-intensive chat model. The provided script and raw output can be found on GitHub, along with a 2-minute guide for testing GPU utilization on individual setups.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.