Only What I Asked
This is a submission for the Kaggle Benchmarking Challenge . I kept running into the same thing with coding models. I'd ask for one small edit and they'd "help" with a pile of other stuff. Fix the redis tag — sure, but also bump nginx. Change a weekday — and Americanize every British spelling while you're at it. Fix a typo in a comment — and rename the misspelled variable that three callers still…
This report examines a benchmark designed to test how well language models follow a single specific request while ignoring all other changes. The study ran 22 test cases across six different models: Claude Opus 5, Sonnet 5, Haiku 4.5, GPT-6 Astra, GPT-5.5, and GPT-5.4 nano. The models were evaluated in two conditions: "plain" and "warned," where the latter explicitly instructed the model not to make any changes other than the one requested.
Across all models, the "plain" condition resulted in 22 out of 22 correct completions, demonstrating near-perfect adherence to the single task. However, the "warned" condition yielded slightly varied results. GPT models performed consistently at 22/22, while Claude Opus 5 and Sonnet 5 also achieved a perfect score. Haiku 4.5, GPT-5.4 nano, and DeepSeek-R1 showed more variation, completing 21/22, 20/22, and 4/22 respectively.
The reason behind these discrepancies lies in how each model handles unrelated changes, with some models inadvertently altering code or text beyond the single requested modification.
Notable issues included Nano incorrectly modifying a constant due to its "plain" instruction, DeepSeek-R1 generating extraneous code blocks in its reasoning, and GPT-5.4 nano's checker capturing multiple tagged blocks instead of just the intended one. These findings highlight the challenges of precise instruction following, especially when models combine reasoning with text generation.
The study concludes that while explicit instructions can improve adherence, they do not guarantee flawless execution, particularly in complex, multi-step scenarios. Future work will explore alternative evaluation metrics and model architectures to better assess instruction following capabilities.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
