{
  "id": 9840788,
  "title": "Why Does Your Local Model Crash at 32k Tokens?",
  "url": "https://urgent.news/2026/09/25/why-does-your-local-model-crash-at-32k-tokens",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-25T21:08:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/eryk_kubiak_c9663554a1ff4/why-does-your-local-model-crash-at-32k-tokens-46ed"
  },
  "original_language": "en",
  "account": "The crash experienced by users when generating long conversations using large language models is not easily explainable. Despite checking the model size against the available VRAM and confirming it fits, the model suddenly crashes after generating several thousand tokens. The weights of the model do not grow during the conversation, so there must be another reason for the crash.\n\nMost guides suggest budgeting only one number: the model's weight size. They ignore a second, independent variable that grows with every token generated - the KV cache. This cache grows as the model reads and writes tokens throughout the conversation. It is not fixed at load time, unlike the model weights, and thus remains a hidden consumer of memory during the entire conversation.\n\nWhen a user loads a 7B model with 4-bit quantization, the model size is approximately 4GB. This number is fixed at load time and remains constant throughout the conversation. However, the KV cache grows with every token generated, and it is not accounted for in the initial memory budgeting process. The crash occurs when the KV cache reaches its limit, and it is not related to the model weights or the prompt content.\n\nTo avoid such crashes, developers should budget not just for the model weights, but also for the KV cache. The KV cache's size depends on the model's architecture and grows linearly with the length of the conversation. By considering both the weight size and the KV cache size, developers can ensure that their local setup has sufficient VRAM to handle long conversations without crashing.",
  "summary": "In this video: 0:00 The Crash Nobody Can Explain 0:18 It Loads, It Answers... Then Dies 1:36 Just Match Weights to VRAM 2:44 OOM at 32k Tokens Anyway 4:32 Weights vs KV Cache, the Real Math 9:00 The 4-Bit Quality Cliff 11:15 Why Hosted APIs Never See This You checked the model size against your VRAM, confirmed it fit, and watched it run fine. So why did it crash forty minutes into a long…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}