{
  "id": 20127737,
  "title": "Running a dense 33B agent model on one 24 GB GPU: the offload trap and the empty reply",
  "url": "https://urgent.news/2026/11/15/running-a-dense-33b-agent-model-on-one-24-gb-gpu-the-offload-trap-and",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-15T06:38:34.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/inferencekey/running-a-dense-33b-agent-model-on-one-24-gb-gpu-the-offload-trap-and-the-empty-reply-4dda"
  },
  "original_language": "en",
  "account": "Two key decisions matter when self-hosting the Nanbeige4.3-33B agent model: selecting a compressed weight copy for your GPU and managing the thinking mode for model reasoning. Using the open-source server llama.cpp on a single RTX 4090 card, the 4-bit file yielded 42 to 44 tokens per second but caused empty replies when employing default thinking mode. This issue arises from dense model architecture, where all 33 billion parameters process each token, causing the GPU to stream the entire 20.8 GB 4-bit file out of VRAM for every token. Partial offloading to CPU layers on a 16 GB card kept answers correct but slowed the model to 1.6 tokens per second. The optimal solution is a smaller file that fits entirely on the GPU, such as bartowski's 3-bit file, which peaked at 56.5 tokens per second on the 4090. Context length, or the amount of text the model processes at once, also impacts speed. A 64K-token prompt allowed the model to locate a secret word at token 61,428, but reduced generation speed to 16 tokens per second. Empty replies occur when using the default thinking mode, as the default chat template opens a thinking block at the end, causing the model to return content in reasoning_content. To avoid this issue, set enable_thinking=false in the chat template kwargs. Other factors to consider include system prompt adjustments, parsing tool calls with the appropriate parser, and testing on various prompt sizes to ensure accurate results.",
  "summary": "To self-host Nanbeige4.3-33B, the Apache-2.0 agent coding model the Nanbeige team (BOSS Zhipin) released on 24 October, two decisions matter most. First, which compressed copy of the weights you load for your card. Second, how you handle thinking mode, where the model reasons before replying: with the defaults, your agent can get an empty answer back. We ran it on one RTX 4090 on our cloud with…",
  "key_points": [
    "33 billion-parameter dense model requires entire 20.8 GB file on GPU for each token",
    "Partial offloading to CPU on 16 GB card slows model to 1.6 tokens per second",
    "3-bit file optimized to fit entirely on GPU, achieving 56.5 tokens per second"
  ],
  "editors_take": "Prioritizing model compression over layer offloading allows dense agent models to maintain high generation speeds on consumer hardware without triggering the empty replies caused by VRAM streaming bottlenecks.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}