Same Model, 13.3% to 38.3%
Two API settings. Same model. Same benchmark. Same task set. 13.3% to 38.3%, using one sixth the output tokens. OpenAI published that result about GPT-5.6 Sol on ARC-AGI-3, and it is the cleanest natural experiment the field has produced on a question I have been arguing from first principles for a year. Nothing about the model changed. Everything that changed was around it. The score is Relative…
OpenAI published a result about GPT-5.6 Sol on ARC-AGI-3, demonstrating a significant improvement in Relative Human Action Efficiency (RHAE) without altering the underlying model. The model's score increased from 13.3% to 38.3%, using only one-sixth of the output tokens. This change was solely due to configuration adjustments, not a new model or prompt.
The official harness used a default behavior of discarding the model's private reasoning after each turn and truncating the context when it exceeded 175,000 characters. This approach forced the model to re-derive rules and strategies on every turn, leading to inefficiency. By retaining reasoning across turns and compacting the history instead of truncating it, OpenAI achieved a 72% reduction in the gap between the model's performance and human performance.
The configuration changes were already used in ChatGPT and Codex, making the benchmark measurements more accurate. The article argues that published benchmark scores often fail to account for the harness design and configuration, leading to inaccurate assessments of the model's capabilities.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
