Notes
🔍

Expected Calibration Error (ECE)

When a classifier says “this email is spam with 90% probability,” you want that number to mean something in the real world. Expected Calibration Error (ECE) is a way to check whether a model’s predicted probabilities line up with how frequently the event actually happens.

What ECE measures

ECE compares confidence (the probabilities your model outputs) to accuracy (the observed frequency of being correct) across groups of similarly confident predictions. The usual recipe is:

  • Split predictions into bins by predicted probability (e.g., 0.0–0.1, 0.1–0.2, …).
  • For each bin, compute:
    • avg confidence: mean predicted probability in the bin
    • empirical accuracy: fraction of correct labels in the bin
  • Take the weighted average of the absolute gaps: |accuracy − confidence|, weighted by how many samples fall in each bin.

Low ECE means “when the model says 70%, it’s right about 70% of the time.” High ECE indicates overconfidence or underconfidence.

Why it matters in practice

ECE matters whenever probabilities drive decisions, not just rankings. Examples:

  • Medical triage: a “95% risk” score should truly correspond to very high risk.
  • Fraud detection: thresholds depend on expected loss; miscalibration can waste investigations or miss fraud.
  • Credit decisions: probability of default feeds pricing and approvals; overconfidence can be costly.

How it’s used (and common pitfalls)

ECE is typically paired with a reliability diagram and used to evaluate post-hoc calibration like Platt scaling, isotonic regression, or temperature scaling. In scikit-learn, calibration workflows commonly involve CalibratedClassifierCV. ECE depends on binning choices (number of bins, equal-width vs equal-frequency), so two reports can differ even for the same model—binning should be stated alongside the score.

Expected Calibration Error (ECE) is a scalar metric that measures how closely a classifier’s predicted probabilities match empirical outcome frequencies. It partitions predictions into confidence bins and computes the weighted average of the absolute gap between average predicted confidence and observed accuracy in each bin. ECE matters because it quantifies probability reliability: low ECE supports risk-sensitive decisions and thresholding, while high ECE signals over/underconfidence even when accuracy is high.

Think of a weather app that says “70% chance of rain.” If, on days it says 70%, it actually rains about 70% of the time, the app is well calibrated. If it rains only 40% of the time, it’s overconfident.

Expected Calibration Error (ECE) is a simple score that tells you how far a model’s confidence is from reality. In supervised learning, many models don’t just predict a label (spam/not spam) — they also give a probability (like 0.9). ECE checks whether “90% confident” really means “right about 90% of the time.” Lower ECE means more trustworthy probabilities.