Self-Hosting Your First LLM: What the Tutorials Skip About GPU Memory
Here is the short version: the model weights are the smallest GPU-memory surprise you'll hit. A 7B model in FP16 needs about 14GB just for weights, but the KV cache — the per-request memory that grows with context length and batch size — is what actually decides whether your setup survives real traffic. Most "run an LLM on your GPU" tutorials load the weights, run one short prompt, and declare…
Self-hosting large language models (LLMs) on a GPU involves more than just downloading and running a pre-trained model. Many tutorials only mention the size of the model weights, but they overlook crucial factors like the key-value (KV) cache and runtime overhead that can quickly exhaust a GPU's memory.
The KV cache is a critical component that stores intermediate results during the model's computation. It scales with the number of model layers, the size of the context (maximum length of input text), and the batch size. For example, a 7B parameter model in FP16 precision needs about 14GB for weights alone. However, when you add the KV cache and other runtime components, the GPU memory requirements can increase substantially.
A 7B model in 4-bit precision might fit on a 24GB GPU with around 20GB left for other memory needs, but this is deceptive. The KV cache for an 8,192-token request could consume up to 1GB alone, and if you serve multiple concurrent requests, that 1GB quickly becomes insufficient. Even with quantization to 4-bit, the KV cache could use up most of the available memory, leaving little room for other operations.
The runtime itself also consumes a significant amount of memory. CUDA and the inference framework each take up several hundred megabytes to a gigabyte of memory, and frameworks like vLLM reserve an extra 1-2GB for efficient management of memory usage. Finally, memory fragmentation can lead to situations where the GPU shows free memory but cannot allocate larger chunks, further reducing the usable memory.
Understanding these memory requirements is crucial for choosing the right GPU, model size, and serving framework. For a first-time self-hosting effort on a single 24GB GPU, starting with a 7-8B model quantized to 4-bit might be the most reliable option. However, as the workload increases, switching to a more robust and optimized serving framework like vLLM can provide better performance and handle higher concurrency more effectively.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — it may contain errors, so check the original before relying on it.
