Why Continuous Batching Made My LLM Inference Engine Slower

A Rust LLM inference engine called mini-vllm-rs experienced slower performance when using continuous batching. Continuous batching involves processing requests together, reusing model weights for more tokens, and increasing throughput. The author tested two concurrent requests on an M2 Mac mini with 16 GB of unified memory, using a 4-bit quantized Qwen2.5 7B model.

Despite the expected improvement, aggregate throughput dropped from 16.8 tokens/s unbatched to 8.7 tokens/s batched, almost halving. After further investigation, the author discovered that a kernel-selection threshold could change generated text, even with greedy decoding. The issue stemmed from the quantized Metal path in Candle, a framework used for backend execution.

The condition checks the number of input rows, and when there is one row, it takes the GEMV path, multiplying the weights by one vector. However, when more than one row is present, it continues into GEMM, which handles multiple input vectors together. Packing two requests changed the input from one row to two, causing the kernel to switch.

GEMM can reuse weights across rows, but the overhead outweighed the benefit at such a small batch size. The author resolved the issue by adding a fallback that split small inputs into rows, ran GEMV for each, and then concatenated the results. They determined that a batch size of four was optimal. However, the author also discovered that speculative decoding with two requests caused a different outcome.

While the overall draft acceptance rate was higher when processed concurrently, the first differing token showed a swapped order of token IDs. This change was enough to flip the greedy choice, affecting the subsequent tokens. The earlier request's presence caused numerical differences in the cached state, which continued even after the batch changed.

By forcing all relevant inputs through GEMV, the author confirmed that the numerical differences were indeed the cause. The author's investigation revealed how a batching decision changed kernel selection, and its effects persisted in cached state long after the batch changed again.