Reliability Diagram
If a model says “there’s a 70% chance this email is spam,” you want that number to mean something in the real world. A reliability diagram is a simple visual check for whether a classifier’s predicted probabilities match what actually happens.
What it shows
A reliability diagram compares predicted probability to observed frequency. You group predictions into bins (for example, 0.0–0.1, 0.1–0.2, …). For each bin, you compute:
- Mean predicted probability in the bin (x-axis)
- Fraction of positives (how often the event truly occurred) in the bin (y-axis)
You then plot these points against the diagonal line y = x. Points on the diagonal mean the model is well-calibrated. Points below the line mean the model is overconfident (predicts probabilities too high); above the line means underconfident.
Why it matters in supervised learning
Accuracy can look great while probabilities are misleading. In credit scoring, a “2% default risk” should correspond to about 2 defaults per 100 similar applicants; pricing, risk limits, and fairness decisions depend on that. In medical triage, poorly calibrated probabilities can cause over-treatment (overconfident high risk) or missed care (underconfident low risk). Reliability diagrams make these failures obvious and guide fixes like Platt scaling or isotonic regression.
How you’ll see it in practice
In scikit-learn, reliability diagrams are commonly made with sklearn.calibration.calibration_curve (and plotted manually). They’re typically paired with a numeric summary like Expected Calibration Error (ECE) or the Brier score, but the diagram is the quickest way to see where calibration breaks.
A Reliability Diagram is a calibration plot that compares a classifier’s predicted probabilities to observed outcome frequencies by binning predictions and plotting empirical accuracy versus predicted confidence (often with a y=x reference line). It reveals whether probabilities are overconfident or underconfident and where miscalibration occurs. This matters because many supervised-learning decisions (thresholding, ranking by risk, cost-sensitive actions) rely on probabilities being trustworthy, not just accurate.
Imagine a weather app that says “70% chance of rain.” If, across many days with a 70% forecast, it actually rains about 7 out of 10 times, the app is being honest about its confidence.
A Reliability Diagram is a simple chart that checks this idea for an AI model’s probability predictions. It groups predictions with similar confidence (like 10%, 30%, 70%) and compares what the model said to what really happened. If the points fall near a diagonal line, the model is well-calibrated. If they drift above or below, the model is overconfident or underconfident.