Probability Calibration
When a classifier says “this customer has a 90% chance of churning,” you want that number to mean something in the real world—not just “the model feels confident.” Probability calibration is about making predicted probabilities line up with actual frequencies.
What calibration means
A model is well-calibrated if, among all examples it predicts at 0.7 probability, about 70% truly belong to the positive class. Many models are good at ranking cases (who is riskier than whom) but still produce probabilities that are systematically too high or too low. This happens because training objectives and model families focus on separating classes, not on making probability estimates numerically truthful.
How it’s measured and visualized
Calibration is commonly checked with a reliability diagram (also called a calibration curve): predictions are binned (e.g., 0.0–0.1, 0.1–0.2, …) and compared to observed outcome rates. A single-number summary you’ll see is the Brier score (mean squared error of probabilistic predictions) or expected calibration error (ECE). Poor calibration shows up as curves above/below the diagonal, indicating under- or over-confidence.
How calibration is fixed in practice
Calibration is often done post-hoc using a held-out validation set (or cross-validation), mapping raw scores to better probabilities:
- Platt scaling: fits a logistic regression on the model’s scores.
- Isotonic regression: a flexible monotonic mapping (needs more data to avoid overfitting).
- Temperature scaling: common for neural nets; rescales logits with one parameter.
In scikit-learn, this is directly supported via
sklearn.calibration.CalibratedClassifierCVProbability calibration is the property (or post-processing) that makes a classifier’s predicted probabilities match empirical outcome frequencies: among instances assigned 0.8, about 80% should be positive. It is assessed with tools like reliability diagrams and metrics such as expected calibration error (ECE), and improved via methods like Platt scaling, isotonic regression, or temperature scaling. It matters because decision thresholds, risk estimates, and cost-sensitive actions depend on trustworthy probabilities.
Think of a weather app that says “70% chance of rain.” If, across many days like that, it actually rains about 7 out of 10 times, the app is well calibrated. If it rains only 3 out of 10 times, those numbers sound confident but mislead you.
Probability calibration is the same idea for AI predictions. Many models can correctly rank outcomes (who is more likely to default on a loan), but their “80%” or “20%” numbers may not match reality. Calibration adjusts or checks those predicted chances so they behave like real-world odds, which matters for decisions like medical risk, fraud alerts, and safety systems.