From Alexmolas · · 50s
Jev can't be calibrated
Jev's probabilities aren't calibrated for your data.
In brief
Jev's probabilities aren't calibrated for your data. Jev's outputs should be treated as scores, not probabilities, due to uncalibrated predictions across different data distributions. Originally reported by Alexmolas.
Jev's usefulness without training data
Jev works without training data for classification tasks, unlike fine-tuned BERT which requires data collection.
Jev's uncalibrated probabilities
TypeSafe claims Jev uses RLCD for calibrated probabilities, but its outputs may not be calibrated for your specific data distribution.
Calibration definition and issue
A model is calibrated when the true probability matches the predicted probability, but Jev may be calibrated for one company's data but not another's.
Jev's coin experiment failure
Jev incorrectly states a fair coin lands heads with probability 0.92, showing its probabilities lack true calibration.
Recalibration solution
Recalibrating Jev with a few hundred labeled examples from your data can fix probability calibration issues.
The main idea
Jev's outputs should be treated as scores, not probabilities, due to uncalibrated predictions across different data distributions.
