Jev Can't Be Calibrated

Jev is a system that TypeSafe has claimed can produce calibrated decisions, but I argue that this isn't generally true. Instead of generating text, Jev takes unstructured input and returns typed decisions with associated probabilities. While TypeSafe says Jev's probabilities are calibrated, I contend this is not the case in general.

Calibration means that for any predicted probability p, the true probability of the positive class given that prediction is also p. However, a model can be calibrated on one data set but not another. Different companies may define the same concept like spam differently, leading to different data distributions. Even if Jev is trained to be calibrated on its own data, its probabilities may not remain calibrated when applied to a different production dataset.

In fact, there's evidence Jev's probabilities can be dramatically off. For example, it claimed a fair coin lands heads with a 92% probability, which is clearly wrong. This suggests Jev's reported probabilities may not accurately reflect the true underlying probabilities.

If you need calibrated probabilities, you'll need to recalibrate Jev's outputs on your own data. This is relatively cheap - a few hundred labeled examples from your actual data can be enough to perform calibration. However, if your system depends on the exact probabilities, like setting thresholds or combining Jev's outputs with other models, you should verify the calibration on your specific data before relying on it.