Paired t-Test

When you compare two models, the tricky part is that their scores aren’t independent: they’re usually measured on the same examples, folds, or datasets. A paired t-test is a simple way to ask: “Is the average difference between these two matched measurements real, or just noise?”

What it is and how it works

A paired t-test compares two methods using paired observations—each pair comes from the same “unit” (for example, the same cross-validation fold). Instead of testing two separate means, it tests whether the mean of the differences is zero. Mechanically:

Compute a difference per pair: d_i = score(A)_i − score(B)_i.
Estimate the mean difference and its standard error.
Use a t-statistic to get a p-value (and often a confidence interval) for whether the mean difference is nonzero.

Where it shows up in supervised learning

Common use: comparing two classifiers’ accuracy across the same k folds, or comparing MAE for two regressors on the same set of time-based splits. In Python you’ll often see SciPy used like:

from scipy.stats import ttest_rel
t, p = ttest_rel(scores_model_a, scores_model_b)

Why it matters (and what can go wrong)

Because the measurements are paired, the test “subtracts out” fold-to-fold difficulty, giving a tighter comparison than an unpaired test. But it relies on assumptions: the differences should be roughly normal and (crucially) independent across pairs. With k-fold CV, folds share training data, so the naive paired t-test can be overconfident. In those cases, practitioners use alternatives like corrected resampled t-tests, nested CV, or nonparametric tests (e.g., Wilcoxon signed-rank) when assumptions are shaky.

A Paired t-Test is a hypothesis test that assesses whether the mean of within-pair differences between two related measurements is zero, assuming the differences are approximately normally distributed. In supervised learning, it compares two models under matched conditions (e.g., per-fold scores from the same cross-validation splits or per-example losses) to determine whether an observed performance gap is statistically credible rather than sampling noise.

Imagine you taste two versions of the same coffee on the same morning—one with sugar, one without—and you rate both. Because the ratings come from the same person in the same situation, it’s a fair “apples-to-apples” comparison. A Paired t-Test is a simple statistical check that asks: are the differences between these matched pairs big enough that we should believe they’re real, not just random wiggles?

In supervised learning, it’s used to compare two models (or one model before vs. after a change) on the same test cases or the same cross-validation splits, to see if one truly performs better.