Serving Groot-19B-A3B at 256K on one RTX 5090: the memory budget and the thinking trap
GrootTeam says its Groot-19B-A3B holds a 256K-token context on one RTX 5090, and labels that figure an estimate from memory sizes. A token is a word piece, and the context is how much text the model can see at once. We wanted a measurement, so we deployed the 20.9 GB FP8 build (weights stored in 8 bits to halve their size) with vLLM 0.32.0, the open-source server that runs the model, on one RTX…
Groot-19B-A3B, a large language model, can process a 256K-token context on a single RTX 5090 GPU. To test the model's memory budget, researchers deployed the 20.9 GB FP8 build with vLLM 0.32.0, which ran the model on an RTX 5090 cloud server. After running the test, they found that the model required a 32 GB budget for optimal performance.
The weights of the model take up 19.7 GiB of the 32 GB budget, leaving 6.8 GiB for the model's KV cache, which is capable of holding 307,024 tokens. The researchers noted that the model's reasoning process contributes to the token count, as each reply includes thinking tokens. They suggested setting thinking off by default to avoid this issue.
The authors emphasized the importance of measuring cache tokens available after weights load, peak memory usage on the longest prompt, and time to first token on a long prompt.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
