Notes
🔍

Margin of Error

When you report a number from data—an average exam score, a survey percentage, a model’s accuracy—you’re really saying “this is my best guess, based on a sample.” The margin of error is the familiar “wiggle room” that tells you how far that guess might reasonably be from the true value.

What it means (technically)

The margin of error is the half-width of a confidence interval. If an estimate is 72 and the margin of error is 3 (at 95% confidence), you’d report 72 ± 3, meaning the 95% confidence interval is [69, 75]. In many common cases, it’s computed as:

margin of error = (critical value) × (standard error)

The critical value comes from a distribution (often a z- or t-distribution) and reflects the chosen confidence level (e.g., 95%). The standard error captures sampling variability and usually shrinks as sample size grows.

Intuition and everyday examples

  • Polling: “52% support ± 4%” means the plausible range is roughly 48%–56% (at the stated confidence level).
  • House prices: If the mean sale price is $410k ± $15k, the data supports a range rather than a single “true” average.
  • Medical measurements: An estimated effect size ± margin of error communicates how precisely the effect was measured.

Why it matters in AI/ML

In ML, point estimates can be misleading without uncertainty. A model with 90% accuracy might not be meaningfully better than one with 88% if their confidence intervals overlap. Margins of error show up when:

  • Reporting evaluation metrics from finite test sets (often via bootstrap confidence intervals).
  • Running A/B tests for ranking, ads, or UI changes.
  • Estimating calibration or conversion rates where sample sizes vary.

Margin of Error is the maximum expected difference between a sample estimate and the true population value at a chosen confidence level. It is typically computed as a critical value (e.g., from a normal or t distribution) times the estimate’s standard error, and it defines the half-width of a confidence interval. In AI/ML, it helps quantify uncertainty in reported metrics or A/B test effects; e.g., accuracy 0.84 ± 0.02.

Imagine you taste one spoonful of soup to judge the whole pot. You can give a best guess (“it’s salty”), but you also know your guess could be a bit off because you didn’t taste everything. The margin of error is that “could be off by about this much” amount.

In statistics (and in AI/ML when we estimate things from data), we often measure something using a sample, not the entire population. The margin of error tells how far the true value might reasonably be from our estimate. A smaller margin of error means a more precise estimate; a larger one means more uncertainty.