Goodness-of-Fit Test
A model can look “reasonable” and still be wrong in important ways. A goodness-of-fit test is a way to check whether the pattern in your observed data is consistent with what a statistical model (or claimed distribution) says should happen.
What it tests (the core idea)
A goodness-of-fit test is a hypothesis test where the null hypothesis (H₀) says: “the data come from this specified distribution/model.” You compare what you actually observed to what the model would predict, and compute a test statistic that measures the mismatch. If the mismatch is too large (small p-value), you reject H₀ and conclude the model is not a good fit.
Common tests you’ll see
Different settings call for different tools:
- Chi-square (χ²) goodness-of-fit: for counts in categories (observed vs expected frequencies).
- Kolmogorov–Smirnov (KS) test: compares an empirical distribution to a continuous reference distribution (often used for 1D continuous data).
- Anderson–Darling: like KS but puts more weight on the tails (useful when extremes matter).
Practical examples
- Retail: Do daily purchase counts by hour match a Poisson model, or are there “bursty” periods the model misses?
- Education: Do multiple-choice answers follow the expected proportions (e.g., random guessing vs biased choices)?
- Healthcare: Do lab measurement errors look normally distributed, or do heavy tails suggest outliers or instrument issues?
Why it matters in AI/ML
Many ML steps quietly assume distributions: noise is Gaussian, residuals are well-behaved, class probabilities are calibrated, or synthetic data matches real data. A failed goodness-of-fit can signal model misspecification, data drift, or unreliable uncertainty estimates—leading to poor predictions and brittle decisions. You’ll encounter these checks in workflows using SciPy (scipy.stats) and statsmodels.
Goodness-of-Fit Test is a hypothesis test that checks whether observed data are consistent with a specified probability distribution or model (the null hypothesis). It quantifies the mismatch between observed and expected frequencies or values, producing a p-value to judge plausibility. In AI/ML, it helps validate distributional assumptions used in probabilistic models and simulations. Example: a chi-square goodness-of-fit test to see if class counts match an expected categorical distribution.
Imagine you have a bag of candy that’s supposed to contain 50% red and 50% blue. You grab a handful and count the colors. A goodness-of-fit test asks: “Do these counts look close enough to the promised mix, or are they too different to be explained by normal randomness?”
In statistics (and in AI/ML), it’s a way to check whether real data matches a model you assume—like whether website clicks follow a certain pattern, or whether a model’s predicted probabilities match what actually happens. If the mismatch is large, it suggests the model may not fit the data well.