Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess

Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess

Senior SREs are often inundated with information about AI models but may not know how to determine the correct node size to host them. This article aims to help with that. It covers estimating GPU memory requirements based on either the model’s parameter count or concurrent requests requirement. After reading, you should be able to confidently select a GPU family.

Background

AI models reside in GPU memory (VRAM), and engines like vLLM manage that memory efficiently. However, the model isn’t the only consumer of VRAM. Other components include:

• Model weights (fixed cost, loaded once)

• KV cache (working memory for in-flight requests)

• Activations (temporary tensors of a forward pass) and CUDA/framework overhead

Key Memory Consumers

• Model weights: consumed once, never shrinks

• KV cache: determines throughput (more cache = more concurrent requests)

• Activations and framework overhead: measured by vLLM at startup

Steps to Determine GPU Sizing

1. Choose a model: The process depends on the model's parameter count. Qwen2.5-7B-Instruct-AWQ will be used as an example.

2. Analyze the spec sheet: The spec sheet provides:

• Parameter count (~7.6 B)

• Quantization (4-bit AWQ)

• KV heads and head dimension

3. Calculate VRAM for weights: Roughly, weights ≈ parameter_count × bytes_per_parameter. Bytes per parameter vary by precision:

• fp16 / bf16: ~2 bytes per parameter → ~15.2 GB

• int8: ~1 byte per parameter → ~7.6 GB

• AWQ (4-bit): ~0.5 bytes per parameter → ~3.8 GB

4. Determine VRAM for KV cache: Subtract weights and overhead from total VRAM. For example, with 24 GB total VRAM:

• Weights: ~5.6 GB

• Remaining for KV cache: ~13.76 GB

• Assuming 57 KB per token, this leaves ~258,000 tokens → ~258 concurrent requests

Conclusion

The same model on an A10 GPU with 24 GB VRAM can support approximately 258 concurrent requests using the AWQ 4-bit variant. This demonstrates how precise GPU sizing can make a significant difference in serving capacity.