Same GPU, Same Model, Different Cost per Token: Benchmark Qwen3.8-27B on Your Own Prompts

When evaluating the cost-effectiveness of renting a GPU for running an open model, it is essential to calculate the cost per million output tokens. This metric is determined by dividing the hourly price by the tokens per second multiplied by 3,600 and then multiplied by 1,000,000. The hourly price is straightforward to obtain, but tokens per second can be challenging to determine accurately, as it depends on the input given to the model.

Published numbers often provide limited insight into this aspect. For the Qwen3.8-27B model on a single RTX PRO 6000 Blackwell (96 GB), the most comprehensive public benchmark was found in EcoHash's speculative-decoding benchmark. The benchmark utilized the FP8 checkpoint on vLLM 0.27.1, with the model's built-in multi-token-prediction (MTP) head enabled.

The model underwent a sweep from 1 to 128 concurrent requests, each receiving 150-token replies. The headline figure is not the most revealing aspect, but what stood out was the inconsistent results when different prompt sets were employed. The latency budget consisted of a 95th-percentile time to first token of 1.5 seconds or less and a 95th-percentile time per output token of 30 milliseconds or less.

The highest concurrency that remained within these parameters varied based on the prompt set. For instance, the "Chat" prompt set reached 32 concurrent requests, the "Math" prompt set achieved 48 concurrent requests, and the "Mixed pool" under the API gateway reached 64 concurrent requests. The cost per 1M output tokens ranged from $0.27 to $0.41, depending on the prompt set.

The disparity in cost per token can be attributed to the draft head of the model, which makes predictions about several tokens ahead of time and checks the predictions in a single pass. Open-ended chat prompts tend to receive fewer accepted guesses, resulting in a higher cost per token compared to arithmetic prompts. Additionally, real replies from the model often exceed the 150-token limit, and longer prompts introduce additional prefill time, affecting the overall cost calculation.

To ensure accurate cost estimations, it is recommended to measure your own GPU performance. The process involves renting an RTX PRO 6000 Blackwell GPU, preferably from a provider that offers pricing at $1.89 per GPU-hour, billed per second. Once the machine is set up, verify that it is indeed the RTX PRO 6000 Blackwell with approximately 96 GB of memory and a compute capability of 12.0.

The benchmark configuration should follow EcoHash's deployment guide, employing the vLLM server with specific flags and settings. Bfloat16 should be used instead of float16, the quantization parameters should be omitted as the checkpoint is already FP8, and the qwen3_xml tool parser should be utilized. The model length limit should be increased to 32768 to accommodate longer prompts, as the benchmark will reveal the extent of the limitation.

By pulling a set of real prompts from your logs, running the benchmark script with varying concurrency levels, and analyzing the results, you can determine the highest concurrency level that fits within your latency budget and calculate the cost per 1M output tokens.