Urgent.News

What's breaking now, across thousands of outlets.

More in Tech

Can vLLM Run GGUF? Yes — on GPU Only

Originally published at mrsaynothing.dev . Last week a reader asked why their GGUF file 404s inside vLLM on a CPU box while the same file runs fine in Ollama.

  • vLLM supports running GGUF models on GPU only
  • Install vllm-gguf-plugin to serve GGUF model
  • --tokenizer flag required for GGUF model serving

More from Wednesday 23 September →