Notes
🔍

Z-test

A Z-test is a way to decide whether an observed difference is “too big to be explained by ordinary randomness.” It’s one of the classic tools for turning a sample result into a yes/no judgment about a population claim.

What it is
A Z-test is a hypothesis test that uses the standard normal distribution (the “Z distribution”) to compute how many standard errors your estimate is away from what the null hypothesis predicts. That distance is the z-statistic:

  • z = (estimate − null value) / standard error

If |z| is large, the corresponding p-value is small, meaning the data would be unlikely if the null were true.

When it’s appropriate
Z-tests are most justified when:

  • The sample size is large (so averages/proportions are approximately normal via the Central Limit Theorem), and
  • The population standard deviation is known, or you’re testing a proportion where the standard error is estimated from the null.

In practice, when the variance is unknown and samples are small, a t-test is usually preferred.

Concrete examples
Common uses include:

  • Mean test: A factory claims the average bottle fill is 500 ml (known σ). You sample 200 bottles and test if the mean differs.
  • Proportion test: An email campaign historically gets 4% clicks. After a change, you test whether the new click rate is different.

Why it matters in AI/ML
In ML workflows, Z-tests show up in:

  • A/B testing for model or UI changes (often as a z-test for proportions or differences in means).
  • Feature and metric monitoring, where standardized shifts (z-scores) flag drift.
  • Interpreting whether observed metric gains are likely real versus noise, especially with large evaluation sets.

A Z-test is a hypothesis test that uses the standard normal (Z) distribution to assess whether a sample mean or proportion differs from a null value, typically assuming a known population variance or a large sample (via the central limit theorem). It matters in AI/ML for quick significance checks on metrics and A/B tests. Example: testing whether a model’s click-through rate exceeds a baseline using a large number of impressions.

Imagine you’re tasting a spoonful of soup to decide if the whole pot is too salty. A Z-test is a similar idea: it uses a small sample to check whether something is likely true about a bigger group.

In statistics (and often in AI experiments), a Z-test helps you compare what you observed to what you’d expect if “nothing special is happening” (the usual baseline). It produces a z-score, which tells how far your result is from the expected value in “standard units.” If that distance is big enough, you treat the difference as unlikely to be due to random chance.