When Does Confidence Reveal Problem Difficulty?
While confidence signals are commonly evaluated for their ability to predict a model's own correctness, their relationship with problem difficulty remains less understood. In this work, we investigate a wide range of confidence-related signals and whether they can serve as indicators of problem difficulty across various task domains. We find that perturbation inconsistency, semantic entropy, and…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
Source: arxiv.org