Benchmarking Qwen 3.8 27B on RTX 5090 and beyond — VRAM capacity alone can't overcome severe software and inference engine bottlenecks

Alibaba’s Qwen 3.8 27B open-weight AI model made a splash when it debuted, impressing AI enthusiasts with its strong benchmark results for a model of its size. Measuring around 17GB for its four-bit quantized weights, Qwen 3.8 27B also comes with built-in multimodal capabilities and general aptitude. Naturally, there was a lot of curiosity around whether this model could run on local AI setups using high-end graphics cards like the RTX 5090, RTX 4090, or RTX 3090, as well as the Radeon RX 7900 XTX, Radeon AI Pro R9700, or Arc Pro B70.

However, as with any open-weight AI model hype, the answer is more nuanced than just comparing the model’s weight size to the available VRAM.

While the RTX 5090 features 32GB of GDDR7 memory and 1.8 TB/s of memory bandwidth, seemingly making it suitable for top-tier local inference performance, initial tests revealed issues. Using llama.cpp as a model runner, the processing speeds at long contexts on the RTX 5090 were disappointing, with time-to-first-token taking about 30 minutes and tokens-per-second throughput falling far below expectations. It turned out that llama.cpp was not the ideal model runner for this hardware at the moment.

To address these challenges, we explored using vLLM, a production-grade inference engine more suited for both desktop and data center environments. Even with 64GB of main memory, vLLM required an additional 64GB of swap space to successfully load Qwen 3.8 27B. Although vLLM could be configured to run on a single RTX 5090 with limited success, it had constraints such as a maximum 32K context length. To fully utilize the model’s 262K context length, a single high-memory card or two RTX 5090s were recommended.

Interestingly, using two RTX 5090s significantly improved decode speeds for vLLM, achieving 70-80 tokens per second across the context depth sweep. Enabling Qwen 3.8 27B’s built-in multi-token prediction (MTP) further boosted the decode speeds to 100-110 tokens per second, offering consistent performance that was reasonable for a local setup. However, the cost of this setup was high, reportedly exceeding $13,000.

We also examined the SGLang inference engine on the RTX 5090, which surprisingly performed almost three times faster than vLLM on a single card. This result suggests that SGLang may be a more viable option for local AI setups with a single RTX 5090.