AirLLM 70B inference with single 4GB GPU

AirLLM has made significant strides in reducing the memory requirements for running large language models (LLMs), making it possible to run models of over 70 billion parameters on a single 4GB GPU. This is achieved through a novel approach known as sparse Mixture-of-Experts (MoE) models, which stream one expert at a time instead of loading an entire layer into memory.

The platform supports a wide range of popular models, including Llama 2, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi, among others, without the need for any special setup or additional configurations.

For example, a 70B Llama model can now run on a single 4GB GPU, a feat that was previously impossible. Similarly, models like Qwen 3.1 (405B), DeepSeek-V3 (671B), and Kimi K3 (2.8T) can run on even less memory, with Qwen 3.1 requiring only 8GB of VRAM and Kimi K3 (2.8T) fitting within 3.72GB. These models have been tested and validated on various GPUs, including the RTX 6000 Ada, demonstrating the platform's versatility and broad applicability.

The advancements in AirLLM are a result of several key optimizations. Per-expert streaming is a crucial mechanism that loads only the experts a token actually routes to, significantly cutting down the memory footprint. Additionally, the platform supports various quantization techniques, such as 8-bit and 4-bit, which further reduce the memory requirements while maintaining reasonable accuracy. The latest version of AirLLM also supports FP8 model support, enhancing performance and efficiency.

To use AirLLM, one simply needs to install the library and pass the Hugging Face repository ID or local path of the desired model to the AutoModel function. The platform initializes the model similarly to a regular transformer model, allowing users to leverage its capabilities with minimal effort. The model is decomposed and saved layer-wise during initialization, ensuring that the disk space requirements are manageable.

This innovative approach to model management enables AirLLM to push the boundaries of what's possible with LLMs, making them more accessible and practical for a wider range of applications.