{
  "id": 1837543,
  "title": "Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess",
  "url": "https://urgent.news/2026/08/19/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-19T00:24:15.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/josef_doornink_930b2caf1c/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess-4fe5"
  },
  "original_language": "en",
  "account": "Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess\n\nSenior SREs are often inundated with information about AI models but may not know how to determine the correct node size to host them. This article aims to help with that. It covers estimating GPU memory requirements based on either the model’s parameter count or concurrent requests requirement. After reading, you should be able to confidently select a GPU family.\n\nBackground\nAI models reside in GPU memory (VRAM), and engines like vLLM manage that memory efficiently. However, the model isn’t the only consumer of VRAM. Other components include:\n• Model weights (fixed cost, loaded once)\n• KV cache (working memory for in-flight requests)\n• Activations (temporary tensors of a forward pass) and CUDA/framework overhead\n\nKey Memory Consumers\n• Model weights: consumed once, never shrinks\n• KV cache: determines throughput (more cache = more concurrent requests)\n• Activations and framework overhead: measured by vLLM at startup\n\nSteps to Determine GPU Sizing\n1. Choose a model: The process depends on the model's parameter count. Qwen2.5-7B-Instruct-AWQ will be used as an example.\n2. Analyze the spec sheet: The spec sheet provides:\n• Parameter count (~7.6 B)\n• Quantization (4-bit AWQ)\n• KV heads and head dimension\n3. Calculate VRAM for weights: Roughly, weights ≈ parameter_count × bytes_per_parameter. Bytes per parameter vary by precision:\n• fp16 / bf16: ~2 bytes per parameter → ~15.2 GB\n• int8: ~1 byte per parameter → ~7.6 GB\n• AWQ (4-bit): ~0.5 bytes per parameter → ~3.8 GB\n4. Determine VRAM for KV cache: Subtract weights and overhead from total VRAM. For example, with 24 GB total VRAM:\n• Weights: ~5.6 GB\n• Remaining for KV cache: ~13.76 GB\n• Assuming 57 KB per token, this leaves ~258,000 tokens → ~258 concurrent requests\n\nConclusion\nThe same model on an A10 GPU with 24 GB VRAM can support approximately 258 concurrent requests using the AWQ 4-bit variant. This demonstrates how precise GPU sizing can make a significant difference in serving capacity.",
  "summary": "Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess OK, you're a senior SRE, you've been hearing incessantly about AI models, but aren't quite sure how to determine the correct node size to host your model. - If so ... you're in the right place. Part of a series on running vLLM on AKS. Companion piece: How to avoid flapping . GPU infrastructure setup — coming soon. This piece…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}