{
  "id": 9373467,
  "title": "Jev Can't Be Calibrated",
  "url": "https://urgent.news/2026/09/23/jev-cant-be-calibrated",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-23T14:39:57.000Z",
  "source": {
    "name": "Hacker News",
    "slug": "hacker-news",
    "url": "https://www.alexmolas.com/2026/09/23/jev-cant-be-calibrated.html"
  },
  "original_language": "en",
  "account": "Jev is a system that TypeSafe has claimed can produce calibrated decisions, but I argue that this isn't generally true. Instead of generating text, Jev takes unstructured input and returns typed decisions with associated probabilities. While TypeSafe says Jev's probabilities are calibrated, I contend this is not the case in general.\n\nCalibration means that for any predicted probability p, the true probability of the positive class given that prediction is also p. However, a model can be calibrated on one data set but not another. Different companies may define the same concept like spam differently, leading to different data distributions. Even if Jev is trained to be calibrated on its own data, its probabilities may not remain calibrated when applied to a different production dataset.\n\nIn fact, there's evidence Jev's probabilities can be dramatically off. For example, it claimed a fair coin lands heads with a 92% probability, which is clearly wrong. This suggests Jev's reported probabilities may not accurately reflect the true underlying probabilities.\n\nIf you need calibrated probabilities, you'll need to recalibrate Jev's outputs on your own data. This is relatively cheap - a few hundred labeled examples from your actual data can be enough to perform calibration. However, if your system depends on the exact probabilities, like setting thresholds or combining Jev's outputs with other models, you should verify the calibration on your specific data before relying on it.",
  "summary": null,
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}