I Turned the Reasoning Dial to 'High' on 4 Models. It Fixed One Thing and Billed Me for Everything.

Reasoning Dial, a Kaggle benchmark, focuses on the reasoning_effort setting which can be adjusted to none, low, medium, or high. Developers typically choose this setting by intuition, often increasing it for challenging questions and decreasing it when in production. However, the benchmark reveals that this decision can significantly impact accuracy, output tokens, and cost per correct answer.

The benchmark tested four AI models under fixed conditions on three types of tasks: Deduce (logic puzzles), Arith (word problems), and Distract (object counting). Across 7 model-task combinations, high reasoning effort led to a notable accuracy boost for gpt-5.4-mini, from 15% to 97.5%. However, in 7 out of 12 model-task combinations, increasing the reasoning effort resulted in higher costs ranging from 1.5 to 3.4 times for each correct answer, while accuracy remained unchanged.

Furthermore, the benchmark uncovered inconsistencies in model behavior. Some models required up to 14 times more tokens at high reasoning effort, while others would reject low reasoning effort with an HTTP 400 error. Additionally, one model continued to operate at none, suggesting the reasoning dial may not uniformly affect all models.

The benchmark's pre-registration documented its hypotheses, statistical methods, and locked data to ensure rigorous analysis. The total cost for the main run was $4.20. Overall, the findings highlight that the reasoning_effort setting can significantly influence AI model performance, token usage, and cost, despite initial intuition that it might only affect the billing.

FINAL ANSWER: The reasoning_effort setting in AI models can drastically impact accuracy, output tokens, and cost, with high settings sometimes improving accuracy but often inflating costs and causing inconsistent model behavior.