vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

LLMs, or large language models, have the potential to revolutionize various industries, but serving these models can be slow and expensive. vLLM, an open-source library developed by researchers at UC Berkeley, enables fast LLM inference and serving using a new attention algorithm called PagedAttention. By implementing PagedAttention, vLLM achieves up to 24x higher throughput compared to HuggingFace Transformers and 3.5x higher throughput than the previous state of the art, HuggingFace Text Generation Inference.

The key to PagedAttention's improved performance lies in its ability to efficiently manage memory, reducing waste and allowing for better GPU utilization. This results in more sequences being batched together, ultimately increasing throughput. PagedAttention also enables memory sharing, further reducing memory overhead in complex sampling algorithms.

This makes LLM serving more affordable for even small research teams like LMSYS with limited compute resources. LMSYS has successfully integrated vLLM into the FastChat serving backend, enabling the serving of popular models like Vicuna, Koala, and LLaMA to millions of users with high throughput and low latency. The use of vLLM has also significantly reduced operational costs, cutting the number of GPUs used by 50%.