{
  "id": 108792,
  "title": "AirLLM 70B inference with single 4GB GPU",
  "url": "https://urgent.news/2026/08/03/airllm-70b-inference-with-single-4gb-gpu",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-03T11:15:48.000Z",
  "source": {
    "name": "Hacker News Best",
    "slug": "hacker-news-best",
    "url": "https://github.com/lyogavin/airllm"
  },
  "original_language": "en",
  "account": "AirLLM has made significant strides in reducing the memory requirements for running large language models (LLMs), making it possible to run models of over 70 billion parameters on a single 4GB GPU. This is achieved through a novel approach known as sparse Mixture-of-Experts (MoE) models, which stream one expert at a time instead of loading an entire layer into memory. The platform supports a wide range of popular models, including Llama 2, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi, among others, without the need for any special setup or additional configurations.\n\nFor example, a 70B Llama model can now run on a single 4GB GPU, a feat that was previously impossible. Similarly, models like Qwen 3.1 (405B), DeepSeek-V3 (671B), and Kimi K3 (2.8T) can run on even less memory, with Qwen 3.1 requiring only 8GB of VRAM and Kimi K3 (2.8T) fitting within 3.72GB. These models have been tested and validated on various GPUs, including the RTX 6000 Ada, demonstrating the platform's versatility and broad applicability.\n\nThe advancements in AirLLM are a result of several key optimizations. Per-expert streaming is a crucial mechanism that loads only the experts a token actually routes to, significantly cutting down the memory footprint. Additionally, the platform supports various quantization techniques, such as 8-bit and 4-bit, which further reduce the memory requirements while maintaining reasonable accuracy. The latest version of AirLLM also supports FP8 model support, enhancing performance and efficiency.\n\nTo use AirLLM, one simply needs to install the library and pass the Hugging Face repository ID or local path of the desired model to the AutoModel function. The platform initializes the model similarly to a regular transformer model, allowing users to leverage its capabilities with minimal effort. The model is decomposed and saved layer-wise during initialization, ensuring that the disk space requirements are manageable. This innovative approach to model management enables AirLLM to push the boundaries of what's possible with LLMs, making them more accessible and practical for a wider range of applications.",
  "summary": "Article URL: https://github.com/lyogavin/airllm Comments URL: https://news.ycombinator.com/item?id=49154228 Points: 185 # Comments: 75",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}