{
  "id": 20050978,
  "title": "Are your model's probabilities calibrated? Expected calibration error in NumPy",
  "url": "https://urgent.news/2026/11/14/are-your-models-probabilities-calibrated-expected-calibration-error",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-14T21:07:15.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/quantgeodesicpartners/are-your-models-probabilities-calibrated-expected-calibration-error-in-numpy-d9"
  },
  "original_language": "en",
  "account": "A recent Hacker News comment highlighted that language model token probabilities are not well-calibrated for making decisions. The author explains how to measure this calibration using a technique called expected calibration error (ECE). By dividing predictions into equal-width bins and comparing the average stated probability with the actual outcomes, one can calculate the ECE. The code snippet provided calculates the ECE using the NumPy library in Python.\n\nThe author then compares the ECE and Brier score of four different models: one that is overconfident, one that always reports the base rate, one that is perfectly calibrated, and one that matches the true chances. The overconfident model performed poorly, while the model that always reports the base rate had an ECE of zero. The model that matches the true chances had the lowest ECE and Brier score, indicating the best calibration.\n\nThe author stresses the importance of reporting both ECE and Brier score, along with a reliability diagram, to assess a model's calibration and sharpness. They also mention that ECE can be affected by the number of bins chosen and provide a link to a worked solution for those who wish to implement and test the ECE calculation themselves.",
  "summary": "There was a comment on Hacker News this week saying LLM token probabilities are badly calibrated for making decisions. I think that's right, and it's also something you can measure yourself in about fifteen lines, so here's how I'd do it. The usual number is expected calibration error. You take a set of predictions where you already know the outcome and put the stated probabilities into m…",
  "key_points": [
    "Expected calibration error (ECE) measures calibration of model probabilities",
    "NumPy library used to calculate ECE in Python code snippet",
    "Perfectly calibrated model has lowest ECE and Brier score among four tested"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}