How Much GPU Memory Do You Really Need to Run an Open LLM Locally?

To run an open-source large language model (LLM) locally, you need to consider three key factors: model size, memory bandwidth, and the size of the key-value cache. Model size is determined by the number of parameters in the model, while memory bandwidth is the rate at which data can be read from and written to memory. The key-value cache is an additional memory requirement for each token processed during attention computations.

Using the official specifications for twelve popular open models, we can calculate the required GPU memory for each model. For instance, Mistral Small 3.2 with 24 billion parameters needs 58 GB in 16-bit precision, 29 GB in 8-bit, or 14 GB in 4-bit. Qwen3.8 27B, with 27.8 billion parameters, requires 67 GB in 16-bit precision, 33 GB in 8-bit, or 17 GB in 4-bit. These calculations give us a baseline for the minimum memory needed to run each model.

However, these are just the starting points. The actual memory requirements may increase significantly based on the context length and the number of concurrent users. The key-value cache size depends on the number of layers, KV heads, and head dimension of the model. For example, Mistral Small 3.2 has a 160 KiB KV cache per token in 16-bit precision, which can grow to 21.5 GB at a 128k-token context.

When planning to deploy an open LLM locally, it's crucial to consider both the weight size and the cache size. While batch processing can help distribute the load across multiple requests, it's essential to account for the peak demand to ensure smooth performance. To do this, you can calculate the required tokens per second by dividing the total monthly request volume by the seconds in a month.

For example, if you expect 200,000 requests per month with an average of 225 tokens per request, you would need to support approximately 23 tokens per second on average.

Keep in mind that real-world performance may differ from theoretical ceilings due to factors like kernel inefficiencies, cache reads during KV computations, and communication overhead when using multiple GPUs. Therefore, it's advisable to benchmark your specific setup to determine the actual throughput achievable with your hardware configuration.

In summary, when assessing how much GPU memory you need for an open LLM, consider the model parameters, memory bandwidth, and key-value cache requirements. Apply the formulas provided in the article to various open models, and always test your system with real-world workloads to ensure it meets your performance expectations.