A confidence number your model reports is not a measurement
A confidence number your model reports is not a measurement The prompt line is almost universal: Answer the question, and also report your confidence as a number between 0 and 1. The model obliges. {"answer": "...", "confidence": 0.92} comes back, and somewhere downstream the code does the obvious thing: if result [ " confidence " ] > 0.8 : apply ( result [ " answer " ]) # auto-accept else :…
A confidence value generated by a model is not a genuine measurement. It is produced alongside the answer during the same forward pass, without any reference to the actual outcome. Measured numbers, on the other hand, are obtained by recording the number and the corresponding result, then comparing them in a database. This distinction is not merely semantic; it is crucial for how the number behaves and what can be derived from it.
Simply asking the model to be more calibrated does not address the fundamental issue, as it only alters the distribution of reported numbers without establishing the necessary recorded outcome. To determine whether you are dealing with a decorative confidence number or a measured probability, examine the shape of the distribution.
Decorative numbers tend to be high and narrow, showing little change when the problem becomes harder. Measured numbers, however, fluctuate more depending on the difficulty and the actual evidence involved. One practical way to distinguish between the two is to pose the same question with different prompts and observe the change in the confidence value.
Measured numbers will adjust based on the difficulty, while decorative numbers will remain relatively constant. A more reliable method is to check if the logs contain the outcome column. If they do not, the number is merely a stylistic choice rather than a verified probability. To improve the situation, make the judgement-making process a separate component with a contract, and log each decision along with its confidence value, inputs, and the action taken.
This allows for tracking the per-band hit rates and ensures that the confidence numbers are based on actual outcomes rather than just sounding convincing.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.