Can vLLM Run GGUF? Yes — on GPU Only
Originally published at mrsaynothing.dev . Last week a reader asked why their GGUF file 404s inside vLLM on a CPU box while the same file runs fine in Ollama. Short answer, before anything else: yes, vLLM runs GGUF — through an official plugin, on GPU only. The plugin is vllm-gguf-plugin , the syntax is repo:quant_type , and the moment you try it on a CPU you are outside the supported hardware…
vLLM now supports running GGUF models, but only on GPU hardware. The official plugin vllm-gguf-plugin allows this functionality. To serve a GGUF model, first install the plugin, then point vLLM to the model using the appropriate syntax. The --tokenizer flag is required and recommended by the official docs, as converting the tokenizer is time-consuming and unstable.
The reason vLLM refuses to run GGUF on CPU is due to architectural limitations, as vLLM is a serving engine and not a laptop toy. The plugin is not yet fully optimized and has some limitations, such as narrower quant coverage and no mmap-style lazy loading. vLLM is best suited for one GPU and many users, while llama.cpp is better for one user and any hardware.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
