Qwen 3.8-Max and Claude Opus 5 show why raw benchmark scores don't predict the bill

Alibaba unveiled Qwen 3.8-Max recently, positioning it as the second-best model after Claude Fable 5. However, an independent benchmark using the preview version placed Qwen 3.8-Max in the middle of the pack, with its default setting performing worst. This discrepancy highlights the impact of token and time budgets on model performance, which are often overlooked in headline benchmark scores.

To accurately assess model costs, two adjustments are necessary. First, metric selection should focus on cost per successful task, accounting for both successful and failed attempts. Second, time and token budgets must be explicit criteria, not hidden details. Price per token alone no longer predicts costs, as reasoning models like Qwen require more thinking tokens to arrive at an answer.

Models that consume most of their token allowance on reasoning may encounter token caps prematurely, resulting in empty results despite the completion of a full run. To properly evaluate agent spend, a cost-per-success metric should be used, considering all spent resources, including unsuccessful attempts. Budget exhaustion and verifier failures are distinct events requiring separate accounting.

Early attempts to measure these costs have shown that timeouts dominate unresolved runs in benchmarks, accounting for 79% of failures in a recent study. This emphasizes the importance of differentiating between budget exhaustion and wrong answers when evaluating model performance. Several entities have already started measuring cost per successful task, including VulcanBench and Long-Horizon-Terminal-Bench.

These benchmarks reveal significant variations in cost and accuracy across models and task combinations. For instance, GPT-5.4 costs approximately $26 per task but achieves a lower pass rate compared to Grok 4.5 at about $11. As vendors increasingly adopt cost-per-success metrics, it is crucial to include failure reasons in every agent run, distinguishing budget exhaustion, verifier failures, and harness errors. This will enable more accurate tracking and optimization of model performance.