Running LLMs Without a GPU: What's Actually Possible

Running large language models (LLMs) on a CPU without a dedicated graphics processing unit (GPU) is feasible, but the performance varies depending on the hardware and the model size. Last year, a friend in Dubai wanted to run a small AI model locally on his three-year-old office laptop with 16 GB of RAM and no discrete GPU. Using a 7B-parameter model, he managed to run it at approximately 11 tokens per second, which was adequate for a private assistant that didn't require serving a large number of users.

The confusion surrounding the necessity of GPUs for local AI arises from conflating two different workloads: training and inference. Training a model is compute-intensive and requires powerful GPUs, while inference, or running the model to generate an answer, is primarily memory-bound. Modern processors have neural accelerators that can help speed up inference on CPUs.

In actual measurements, token generation rates on consumer hardware are as follows:

- A 2021 office laptop with 16 GB RAM can run a Llama-3.1-8B model with quantization Q4_K_M at 5 GB memory speed, achieving 10-12 tokens per second.

- An Apple M2 or M3 laptop, with its unified memory, can run the same model at 25-35 tokens per second.

- A recent laptop with an NPU (on-device neural processing unit) can handle a 7-8B model, also at about 15-30 tokens per second.

For desktop setups, a recent CPU with 16 cores and no GPU can run an 8B model with Q4_K_M at 5 GB memory speed, delivering 15-20 tokens per second. Increasing the RAM to 32 GB can improve performance for a 14B model to 8-12 tokens per second. On desktops with iGPUs, there is a modest gain compared to using the CPU alone.

Token generation is memory-bound, while prompt processing is compute-bound. Short prompts with long answers perform well on CPUs, while long prompts with short answers may feel sluggish due to the time taken to process the input. To optimize performance, it is advisable to cap context and keep inputs tight.

The crucial components of a CPU inference setup include:

1. Quantized models (e.g., GGUF format) that reduce memory and bandwidth requirements by shrinking the model's weights from 16-bit floats to 4-bit integers.

2. llama.cpp, a pure C/C++ reference runtime that efficiently runs on CPUs and can offload layers to GPUs or NPUs where available.

3. A server layer, such as the OpenAI-compatible chat/completions API server provided by llama.cpp, which allows existing tools that use the OpenAI API to function with local CPU models.

4. A wrapper like Ollama or llama-cpp-python for model management and a simple command-line interface.

To set up a local CPU inference server, one can install Ollama, pull a quantized model, and serve it using a single command. The resulting endpoint is OpenAI-compatible and can be tested using a simple HTTP request. This demonstrates that local AI is not exclusively reliant on expensive hardware, but rather on the right software stack and optimization techniques.