{
  "id": 8146013,
  "title": "How to Run GGUF Models Locally: Ollama, llama.cpp & vLLM",
  "url": "https://urgent.news/2026/09/18/how-to-run-gguf-models-locally-ollama-llama-cpp-vllm",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-18T01:47:49.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/mrsaynothing/how-to-run-gguf-models-locally-ollama-llamacpp-vllm-59ek"
  },
  "original_language": "en",
  "account": "GGUF files are compressed language model files that hold weights, tokenizer, and model metadata in a single container. They are lightweight and can run on local hardware without downloading additional components. Ollama is a popular local LLM runner that supports GGUF files directly from Hugging Face, making it easy to run models like gemma-3-4b-it-GGUF with a simple command like \"ollama run hf.co/bartowski/gemma-2-9b-it-GGUF:Q4_K_M\". GMUF models are quantized weights that require less VRAM compared to full-precision 16-bit floats, allowing for faster inferences on GPUs and CPUs.\n\nTo run a GGUF model in llama.cpp, you can use the llama-server binary with a modified command such as \"llama-server -m ./gemma-2-9b-it-Q4_K_M.gguf -ngl 99 --port 8080\". The -ngl flag controls how many layers are offloaded to the GPU, with the rest running on the CPU. For one-shot prompting, llama-cli can be used similarly.\n\nChoosing the right quantization level depends on the desired balance between model size and performance. Q4_K_M is a common choice, offering a good compromise between quality and size. Other quantization levels, such as Q8_0 or Q3_K_M, provide even better compression but may result in some quality loss.\n\nWhen considering which runner to use, Ollama is the best choice for a set-and-forget service that runs at boot and is accessible to multiple devices on the network. Llama.cpp is ideal for those who need the latest features and fine-grained control over layer allocation. LM Studio offers a user-friendly desktop GUI for managing models, while vLLM is suited for batched multi-user serving scenarios. Overall, the choice between GGUF and Safetensors formats depends on whether you are training or serving local chat applications.",
  "summary": "Downloaded a .gguf file and wondering how to actually run it? Fastest path: ollama run hf.co/<repo>:Q4_K_M — Ollama pulls the GGUF straight from Hugging Face and serves it. GGUF is the single-file model format that llama.cpp introduced and every local-LLM tool now speaks, so the same file runs in Ollama, llama.cpp, LM Studio, Jan, and (with caveats) vLLM. This guide covers each runner with…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}