Law of Large Numbers
The Law of Large Numbers is the simple idea that “randomness settles down” when you collect enough data. A small sample can look surprising, but as the sample grows, the average becomes more stable and predictable.
What it says (in plain terms)
Suppose you repeatedly measure something that has a true long-run average (like the average height in a city, or the average click-through rate of an ad). The Law of Large Numbers says that the sample mean (your average from the data you collected) will get closer and closer to the expected value (the true average) as the number of observations increases. This doesn’t mean the data stop being noisy—individual points can still vary a lot—but the average becomes less “jumpy.”
Quick intuition
Think of flipping a fair coin. After 10 flips, getting 8 heads isn’t shocking. After 10,000 flips, 8,000 heads would be shocking. Large samples “dilute” streaks and quirks.
Everyday examples
- House prices: The average sale price from 5 homes can be skewed by one luxury listing; from 5,000 homes it’s much steadier.
- Exam scores: A class average from 8 students can swing a lot; from 800 students it’s more reliable.
- Medical measurements: Average blood pressure estimated from a few patients is unstable; larger studies converge toward the population mean.
Why it matters in AI/ML
ML relies on data-based estimates: average loss, accuracy, gradients, and feature statistics. The Law of Large Numbers helps justify why:
- Training metrics become more reliable with more data.
- Mini-batch gradient estimates improve as batch size increases (less variance).
- A/B test results stabilize as sample size grows.
Without it, “learning from data” would feel like chasing noise with no guarantee that averages reflect reality.
Law of Large Numbers states that as the number of independent, identically distributed samples grows, the sample average converges to the true expected value (population mean). It underpins why empirical estimates become reliable with more data, justifying Monte Carlo methods, A/B testing, and many ML training statistics. Example: averaging many stochastic gradient minibatch losses yields a stable estimate of the true expected loss.
Imagine flipping a coin. If you flip it just a few times, you might get lots of heads by luck. But if you flip it hundreds or thousands of times, the results usually settle down close to 50% heads and 50% tails.
That calming-down effect is the Law of Large Numbers: when you repeat something random many times, the average result tends to get closer to the true long-run average.
In statistics and machine learning, this is why using more data often gives more reliable estimates and predictions—random “noise” cancels out, and the real pattern shows through.