Why vLLM and SGLang might not be the best option to run AI agents
vLLM and SGLang are great inference engines. If you serve chat or batch jobs, use them. But AI agents are a different workload, and I wanted to see how well these engines handle it. So I measured it. How an agent talks to the model A coding agent works in turns: It sends the whole conversation to the model. The model answers with a tool call (run a command, edit a file). The agent runs the tool,…
vLLM and SGLang are powerful inference engines suitable for serving chat or batch jobs. However, when it comes to AI agents, they may not be the optimal choice. The author conducted a test to compare how well these engines handle agent workloads.
AI agents communicate with the model in turns, sending the entire conversation history each time. The model then responds with a tool call, after which the agent runs the tool, appends the result, and sends the updated conversation. This process requires the engine to access the key-value (KV) cache for the full conversation history.
If the cache from the previous turn is still in GPU memory, only the new tokens need to be computed. However, if the cache is evicted while the agent is running its tool, the engine must recompute the entire history.
vLLM 0.33.0 has a prefix cache that stores blocks from earlier requests, but it doesn't prioritize them based on agent sessions. It only uses session_id for metadata purposes and evicts the least recently used blocks when memory runs out. On the other hand, SGLang 0.5.23 offers real session management (/open_session, /close_session), but its OpenAI-compatible chat endpoint only passes the session_id, not the session parameters.
The mode that pins the session's KV cache is disabled by default and requires the /generate API. The default eviction policy is LRU, with options like TLRU.
To evaluate the performance of these engines in handling agent workloads, the author created a testing framework in Rust. The test involved running 24 coding-agent sessions from the SWE-rebench-OpenHands dataset simultaneously. The engine maintained each agent's session in GPU memory between turns, shared the common prompt start (system prompt, tool definitions) across sessions, and only admitted new sessions when existing ones fit within memory constraints.
The test compared the Rust session engine with vLLM and SGLang under three different tool wait times: 0.5 seconds, 5 seconds, and 60 seconds.
The results showed that the Rust session engine significantly reduced history recomputation, cutting it to 8-16% of total tokens processed. This optimization led to a 1.31x to 1.45x improvement in overall run time compared to SGLang and vLLM, respectively. Additionally, the first token was produced about 5 times faster with the Rust session engine. However, the gain diminishes when tool waits are longer. With a 60-second wait, the performance difference between engines becomes negligible.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
