Calibration Curve
When a classifier says “this email has a 70% chance of being spam,” you want that number to mean something in the real world. A calibration curve is a simple visual check: do predicted probabilities line up with what actually happens?
What the curve shows
A calibration curve (also called a reliability diagram) compares a model’s predicted probability to the observed frequency of the positive class. You typically:
- Group predictions into bins (e.g., 0.0–0.1, 0.1–0.2, …).
- For each bin, compute the average predicted probability (x-axis).
- Compute the fraction of true positives in that bin (y-axis).
If the model is perfectly calibrated, points fall on the diagonal line y = x. Points below the diagonal mean the model is overconfident (predicts 0.8 but only 0.6 happen). Above the diagonal means underconfident.
Why calibration matters in practice
Calibration matters whenever you act on probabilities, not just rankings. Examples:
- Credit risk: a “5% default risk” should really default about 5% of the time for pricing and capital decisions.
- Medical triage: probability thresholds determine who gets further testing.
- Fraud detection: calibrated scores support consistent alert volumes as conditions change.
A model can have strong ROC-AUC yet poor calibration; it can rank cases well while mis-stating absolute risk.
Tools you’ll see
In scikit-learn, sklearn.calibration.calibration_curve computes the points, and CalibratedClassifierCV applies Platt scaling (sigmoid) or isotonic regression to improve calibration.
A Calibration Curve (reliability diagram) plots a model’s predicted probabilities against the observed event frequencies, typically by binning predictions and comparing mean predicted risk to empirical outcomes. A perfectly calibrated classifier lies on the diagonal, while deviations reveal over- or under-confidence. It matters because many decisions, threshold choices, and downstream risk estimates assume probabilities are trustworthy; poor calibration can make accurate classifiers produce misleading probability outputs.
Imagine a weather app that says “70% chance of rain.” If, over many days when it says 70%, it actually rains about 7 out of 10 times, that app is well calibrated. A Calibration Curve is a simple chart that checks this idea for an AI model’s probability predictions.
It compares what the model says (like “there’s a 20% chance this email is spam”) to what really happens in real life. If the curve stays close to a diagonal “perfect honesty” line, the probabilities are trustworthy. If it bends away, the model may be overconfident or too cautious.