Notes
🔍

Mann-Whitney U Test

The Mann-Whitney U test is a way to compare two independent groups when you don’t fully trust “bell-curve” assumptions. It’s often used when data are skewed, have outliers, or are naturally ordinal (rank-like) rather than truly numeric.

What it does (in plain terms)
Technically, the Mann-Whitney U test is a non-parametric test for whether two groups tend to produce different values. Instead of comparing means (like a t-test), it compares the ranks of all observations after pooling both groups together. If one group consistently has higher ranks, the test statistic (U) becomes extreme, leading to a small p-value.

What hypothesis it’s really testing
You’ll often hear it described as “testing whether medians differ,” but that’s only reliably true when the two groups have similarly shaped distributions. More generally, it tests whether a randomly chosen value from group A is equally likely to be larger than a randomly chosen value from group B (a statement about stochastic dominance).

Practical examples
Common uses include:

  • Comparing house prices in two neighborhoods when prices are heavily right-skewed.
  • Comparing exam scores between two teaching methods when scores have ceiling effects.
  • Comparing patient pain ratings (1–10) between two treatments (ordinal data).

Why it matters in AI/ML
In ML, it’s useful for quick, assumption-light comparisons such as:

  • A/B testing two model variants on per-user losses or latencies (often skewed).
  • Feature screening: do two classes differ in a feature’s distribution without assuming normality?
  • Evaluating whether a change shifts the typical outcome, not just the mean.

You’ll see it in SciPy as scipy.stats.mannwhitneyu.

The Mann-Whitney U Test is a nonparametric test for assessing whether two independent samples tend to come from the same distribution, often interpreted as a difference in central tendency (median) when shapes are similar. It uses ranked data, avoiding normality assumptions, making it useful for skewed or ordinal outcomes common in ML evaluation. Example: comparing AUC scores from two models across datasets when scores are not normally distributed.

Imagine you’re comparing two groups of runners, but you don’t trust the stopwatch readings to be nicely “well-behaved” or evenly spread out. Instead of relying on averages, you line up everyone’s times from fastest to slowest and see which group tends to land closer to the fast end.

The Mann-Whitney U Test does this kind of comparison for two independent groups. It checks whether one group generally has higher or lower values than the other, without assuming the data follows a bell curve. In AI/ML, it’s often used to compare model results (like errors or scores) when the numbers are skewed or have outliers.