acc vs acc_norm: Why Length Bias Skews LLM Eval Scores
Your fine-tune gains three points of acc_norm on HellaSwag and loses two points of acc . Same checkpoint, same harness, same seed. Nothing about the model's commonsense reasoning moved in two directions at once — you changed its average per-token entropy, and one of those two metrics is partly measuring how long the answer string is. This is the acc vs acc_norm problem, and it decides which…
The headline "acc vs acc_norm: Why Length Bias Skews LLM Eval Scores" highlights the difference between two metrics used to evaluate language models (LLMs). "acc" calculates the highest summed log-likelihood of a candidate continuation, while "acc_norm" divides that sum by the byte length of the continuation string. This difference can significantly impact the ranking of candidates, especially when answer lengths vary.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
