How to Run GGUF Models Locally: Ollama, llama.cpp & vLLM

GGUF files are compressed language model files that hold weights, tokenizer, and model metadata in a single container. They are lightweight and can run on local hardware without downloading additional components. Ollama is a popular local LLM runner that supports GGUF files directly from Hugging Face, making it easy to run models like gemma-3-4b-it-GGUF with a simple command like "ollama run hf.co/bartowski/gemma-2-9b-it-GGUF:Q4_K_M".

GMUF models are quantized weights that require less VRAM compared to full-precision 16-bit floats, allowing for faster inferences on GPUs and CPUs.

To run a GGUF model in llama.cpp, you can use the llama-server binary with a modified command such as "llama-server -m ./gemma-2-9b-it-Q4_K_M.gguf -ngl 99 --port 8080". The -ngl flag controls how many layers are offloaded to the GPU, with the rest running on the CPU. For one-shot prompting, llama-cli can be used similarly.

Choosing the right quantization level depends on the desired balance between model size and performance. Q4_K_M is a common choice, offering a good compromise between quality and size. Other quantization levels, such as Q8_0 or Q3_K_M, provide even better compression but may result in some quality loss.

When considering which runner to use, Ollama is the best choice for a set-and-forget service that runs at boot and is accessible to multiple devices on the network. Llama.cpp is ideal for those who need the latest features and fine-grained control over layer allocation. LM Studio offers a user-friendly desktop GUI for managing models, while vLLM is suited for batched multi-user serving scenarios. Overall, the choice between GGUF and Safetensors formats depends on whether you are training or serving local chat applications.