{
  "id": 4550714,
  "title": "Running LLMs Without a GPU: What's Actually Possible",
  "url": "https://urgent.news/2026/08/31/running-llms-without-a-gpu-whats-actually-possible",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-31T02:30:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/mryadavgulshan/running-llms-without-a-gpu-whats-actually-possible-1b7p"
  },
  "original_language": "en",
  "account": "Running large language models (LLMs) on a CPU without a dedicated graphics processing unit (GPU) is feasible, but the performance varies depending on the hardware and the model size. Last year, a friend in Dubai wanted to run a small AI model locally on his three-year-old office laptop with 16 GB of RAM and no discrete GPU. Using a 7B-parameter model, he managed to run it at approximately 11 tokens per second, which was adequate for a private assistant that didn't require serving a large number of users.\n\nThe confusion surrounding the necessity of GPUs for local AI arises from conflating two different workloads: training and inference. Training a model is compute-intensive and requires powerful GPUs, while inference, or running the model to generate an answer, is primarily memory-bound. Modern processors have neural accelerators that can help speed up inference on CPUs.\n\nIn actual measurements, token generation rates on consumer hardware are as follows:\n- A 2021 office laptop with 16 GB RAM can run a Llama-3.1-8B model with quantization Q4_K_M at 5 GB memory speed, achieving 10-12 tokens per second.\n- An Apple M2 or M3 laptop, with its unified memory, can run the same model at 25-35 tokens per second.\n- A recent laptop with an NPU (on-device neural processing unit) can handle a 7-8B model, also at about 15-30 tokens per second.\n\nFor desktop setups, a recent CPU with 16 cores and no GPU can run an 8B model with Q4_K_M at 5 GB memory speed, delivering 15-20 tokens per second. Increasing the RAM to 32 GB can improve performance for a 14B model to 8-12 tokens per second. On desktops with iGPUs, there is a modest gain compared to using the CPU alone.\n\nToken generation is memory-bound, while prompt processing is compute-bound. Short prompts with long answers perform well on CPUs, while long prompts with short answers may feel sluggish due to the time taken to process the input. To optimize performance, it is advisable to cap context and keep inputs tight.\n\nThe crucial components of a CPU inference setup include:\n1. Quantized models (e.g., GGUF format) that reduce memory and bandwidth requirements by shrinking the model's weights from 16-bit floats to 4-bit integers.\n2. llama.cpp, a pure C/C++ reference runtime that efficiently runs on CPUs and can offload layers to GPUs or NPUs where available.\n3. A server layer, such as the OpenAI-compatible chat/completions API server provided by llama.cpp, which allows existing tools that use the OpenAI API to function with local CPU models.\n4. A wrapper like Ollama or llama-cpp-python for model management and a simple command-line interface.\n\nTo set up a local CPU inference server, one can install Ollama, pull a quantized model, and serve it using a single command. The resulting endpoint is OpenAI-compatible and can be tested using a simple HTTP request. This demonstrates that local AI is not exclusively reliant on expensive hardware, but rather on the right software stack and optimization techniques.",
  "summary": "The honest numbers on CPU, NPU, and iGPU inference — what runs, how fast, and when to stop pretending. Last year a friend who runs a small consulting firm in Dubai asked me what GPU he needed to \"run AI locally.\" He had a budget in mind and had already been quoted a four-figure price tag for a workstation. I asked him what he actually wanted to run. A document chatbot over his firm's contracts.…",
  "key_points": [
    "Running LLMs on CPU possible, performance varies by hardware",
    "Token generation rates: 10-12 tokens/sec on 2021 laptop, 25-35 on M2/M3",
    "Desktop CPU with 16 cores can run 8B model at 15-20 tokens/sec"
  ],
  "editors_take": "Running large language models on CPUs without GPUs is feasible with varied performance, and the right software stack and optimization techniques can make local AI accessible without expensive hardware.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}