Are your model's probabilities calibrated? Expected calibration error in NumPy
There was a comment on Hacker News this week saying LLM token probabilities are badly calibrated for making decisions. I think that's right, and it's also something you can measure yourself in about fifteen lines, so here's how I'd do it. The usual number is expected calibration error. You take a set of predictions where you already know the outcome and put the stated probabilities into m…
A recent Hacker News comment highlighted that language model token probabilities are not well-calibrated for making decisions. The author explains how to measure this calibration using a technique called expected calibration error (ECE). By dividing predictions into equal-width bins and comparing the average stated probability with the actual outcomes, one can calculate the ECE. The code snippet provided calculates the ECE using the NumPy library in Python.
The author then compares the ECE and Brier score of four different models: one that is overconfident, one that always reports the base rate, one that is perfectly calibrated, and one that matches the true chances. The overconfident model performed poorly, while the model that always reports the base rate had an ECE of zero. The model that matches the true chances had the lowest ECE and Brier score, indicating the best calibration.
The author stresses the importance of reporting both ECE and Brier score, along with a reliability diagram, to assess a model's calibration and sharpness. They also mention that ECE can be affected by the number of bins chosen and provide a link to a worked solution for those who wish to implement and test the ECE calculation themselves.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
