Notes
🔍

Maximum Calibration Error (MCE)

When a classifier says “this email is spam with 90% probability,” you want that number to mean something in the real world: among many emails scored at 0.9, about 90% should truly be spam. Maximum Calibration Error (MCE) is a way to quantify the worst place where those probabilities stop matching reality.

What MCE measures

MCE looks at how far predicted probabilities are from observed outcomes, but it focuses on the single most miscalibrated region. The usual recipe is:

  • Collect predicted probabilities (e.g., from LogisticRegression, a neural net, or XGBoost) and true labels.
  • Split predictions into bins (like 0.0–0.1, 0.1–0.2, …).
  • For each bin, compute:
    • confidence: the average predicted probability in the bin
    • accuracy: the fraction of positives in the bin
    • gap = |accuracy − confidence|
  • MCE = the maximum gap across all bins.

So if one bin has predictions around 0.8 but only 0.6 of them are actually positive, that bin’s gap is 0.2—and MCE is at least 0.2.

Why it matters in supervised learning

MCE is a “worst-case” calibration check. It matters when decisions depend on reliable probabilities, not just rankings:

  • Medical triage: a badly miscalibrated high-risk band can overload resources or miss urgent cases.
  • Credit or fraud review: thresholds like “send to manual review if risk > 0.7” break if the 0.7 region is unreliable.

Compared with average-style metrics (like ECE), MCE is harsher: it highlights a single problematic slice that could be operationally dangerous.

Maximum Calibration Error (MCE) measures the worst-case mismatch between a model’s predicted probabilities and observed outcome frequencies across probability bins. It is computed as the maximum, over bins, of the absolute difference between average predicted confidence and empirical accuracy (or event rate) in that bin. MCE matters because it exposes the largest local miscalibration, helping assess reliability for high-stakes decisions where a single badly calibrated region is unacceptable.

Imagine a weather app that says “80% chance of rain.” If, over many days with that same 80% forecast, it actually rains about 80% of the time, the app is well calibrated. If it only rains 50% of the time, the app is overconfident.

Maximum Calibration Error (MCE) is a way to measure the worst mismatch like this. You group predictions by confidence (for example, 60–70%, 70–80%), compare what the model “claimed” to what really happened in each group, and then take the biggest gap. A low MCE means the model’s probabilities are trustworthy even in its worst spot.