Running a dense 33B agent model on one 24 GB GPU: the offload trap and the empty reply

Two key decisions matter when self-hosting the Nanbeige4.3-33B agent model: selecting a compressed weight copy for your GPU and managing the thinking mode for model reasoning. Using the open-source server llama.cpp on a single RTX 4090 card, the 4-bit file yielded 42 to 44 tokens per second but caused empty replies when employing default thinking mode.

This issue arises from dense model architecture, where all 33 billion parameters process each token, causing the GPU to stream the entire 20.8 GB 4-bit file out of VRAM for every token. Partial offloading to CPU layers on a 16 GB card kept answers correct but slowed the model to 1.6 tokens per second. The optimal solution is a smaller file that fits entirely on the GPU, such as bartowski's 3-bit file, which peaked at 56.5 tokens per second on the 4090.

Context length, or the amount of text the model processes at once, also impacts speed. A 64K-token prompt allowed the model to locate a secret word at token 61,428, but reduced generation speed to 16 tokens per second. Empty replies occur when using the default thinking mode, as the default chat template opens a thinking block at the end, causing the model to return content in reasoning_content.

To avoid this issue, set enable_thinking=false in the chat template kwargs. Other factors to consider include system prompt adjustments, parsing tool calls with the appropriate parser, and testing on various prompt sizes to ensure accurate results.