GGUF VRAM calculator: the three numbers that decide
Cross-posted from Mr Say Nothing — receipts live there. "Will this quant fit?" is a three-number question, not a feeling: weights + KV cache + overhead . I printed the formulas and shipped them as a calculator you can open from disk — then checked the math against four public GGUF files. Worst miss: 0.04 GB , under one percent. Here is the whole decision on paper, so the allocator never gets to…
The GGUF VRAM calculator reveals that three key numbers determine whether a quant fits within the available VRAM: weights, KV cache, and overhead. The formula calculates weights as parameters multiplied by 1e9, then by the quant bits-per-weight divided by 8, and then by 1.08. KV cache is calculated as twice the number of layers, KV heads, head dimensions, and a factor of 2 bytes.
The overhead is the GPU's reserved resources, estimated between 0.6-1.2 GiB. The calculator, available as a downloadable file, uses real-world GGUF files from Hugging Face to test the accuracy, with an overall drift of just 0.04 GB. The KV cache is often overlooked but grows significantly with context size. For example, Llama-3 8B with 32 layers and 8 KV heads running at 32k context would exceed an 8GB card's capacity.
The calculator highlights the importance of considering KV heads and context length, especially for older models, to avoid out-of-memory errors. It also points out blind spots like MoE sharding, draft models, and the impact of desktop usage on VRAM allocation. The tool is a simple, static page that emphasizes the importance of understanding these three numbers to predict VRAM usage accurately.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
