Friedman Test

When you compare several machine learning models, the tricky part isn’t getting a leaderboard—it’s knowing whether the differences are real or just noise from the particular datasets or folds you happened to use. The Friedman test is a practical way to answer that question without assuming the scores are nicely “bell-shaped.”

What it is and what it tests

The Friedman test is a non-parametric statistical test for comparing three or more methods across multiple matched blocks (for ML, blocks are typically datasets or cross-validation folds). Instead of comparing raw metric values directly, it converts results into ranks within each block (best gets rank 1, next rank 2, etc.), then checks whether the average ranks differ more than you’d expect by chance. The null hypothesis is: all models perform equivalently (their rank distributions are the same).

How it shows up in supervised learning

Common use cases include comparing classifiers by AUC or F1 across many datasets (spam detection, churn, fraud) or comparing regressors by RMSE across multiple forecasting problems. A typical workflow:

• Evaluate each model on each dataset/fold.
• Rank models within each dataset/fold by the metric.
• Run the Friedman test on the rank table.

Why it matters (and what to do next)

If you skip statistical comparison, you can “pick a winner” that only won due to sampling luck. If the Friedman test is significant, it tells you at least one model differs—but not which ones. You typically follow it with a post-hoc procedure such as the Nemenyi test or pairwise tests with multiple-comparison correction. In Python, you’ll encounter this via SciPy’s scipy.stats.friedmanchisquare (with care: it expects matched measurements per block).