{
  "id": 9371129,
  "title": "Can vLLM Run GGUF? Yes — on GPU Only",
  "url": "https://urgent.news/2026/09/23/can-vllm-run-gguf-yes-on-gpu-only",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-23T16:28:44.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/mrsaynothing/can-vllm-run-gguf-yes-on-gpu-only-1hpj"
  },
  "original_language": "en",
  "account": "vLLM now supports running GGUF models, but only on GPU hardware. The official plugin vllm-gguf-plugin allows this functionality. To serve a GGUF model, first install the plugin, then point vLLM to the model using the appropriate syntax. The --tokenizer flag is required and recommended by the official docs, as converting the tokenizer is time-consuming and unstable. The reason vLLM refuses to run GGUF on CPU is due to architectural limitations, as vLLM is a serving engine and not a laptop toy. The plugin is not yet fully optimized and has some limitations, such as narrower quant coverage and no mmap-style lazy loading. vLLM is best suited for one GPU and many users, while llama.cpp is better for one user and any hardware.",
  "summary": "Originally published at mrsaynothing.dev . Last week a reader asked why their GGUF file 404s inside vLLM on a CPU box while the same file runs fine in Ollama. Short answer, before anything else: yes, vLLM runs GGUF — through an official plugin, on GPU only. The plugin is vllm-gguf-plugin , the syntax is repo:quant_type , and the moment you try it on a CPU you are outside the supported hardware…",
  "key_points": [
    "vLLM supports running GGUF models on GPU only",
    "Install vllm-gguf-plugin to serve GGUF model",
    "--tokenizer flag required for GGUF model serving"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}