t-test
A t-test is a way to check whether a difference you see in data is likely to be real, or whether it could easily be explained by ordinary random variation. It’s especially handy when you have a small-ish sample and you’re comparing averages.
What it does (in plain terms)
A t-test evaluates a null hypothesis like “the two groups have the same mean.” It computes a t-statistic, which is essentially:
- difference in sample means (or mean vs. a target value)
- divided by an estimate of the standard error (how noisy that difference is expected to be)
From the t-statistic and the degrees of freedom, you get a p-value: the probability of seeing a difference at least this large if the null hypothesis were true.
Common types you’ll see
Different setups call for different t-tests:
- One-sample t-test: compare a sample mean to a known value (e.g., average exam score vs. 70).
- Two-sample t-test: compare means of two independent groups (e.g., house prices in two neighborhoods).
- Paired t-test: compare “before vs. after” on the same items (e.g., blood pressure pre/post medication).
Why it matters in AI/ML
In ML, t-tests often appear when comparing models or experiments:
- A/B testing feature changes (conversion rate proxies, revenue per user).
- Comparing cross-validation scores between two algorithms (with care about dependence).
- Checking whether a metric improvement is likely more than noise.
It relies on assumptions like approximate normality of errors and appropriate independence; when those don’t hold, results can mislead. In practice you’ll encounter it in SciPy (scipy.stats.ttest_*) and statsmodels.
A t-test is a hypothesis test that uses the t-distribution to assess whether a sample mean differs from a reference value or whether two means differ, especially when sample sizes are small and the population variance is unknown. It is important in AI/ML for validating observed performance differences and experimental effects under uncertainty. Example: testing whether a new model’s mean accuracy across folds is higher than a baseline’s.
Imagine you’re taste-testing two batches of cookies and want to know if one is truly sweeter, or if you just happened to pick a few unusually sweet cookies by chance. A t-test does the same kind of check with data: it compares an average (like the mean score, time, or rating) between two groups, or between a group and a known value, and asks whether the difference looks “real” or likely due to random variation.
In statistics and AI/ML, a t-test is often used to judge whether an experiment or model change (like a new feature) actually improved results, especially when you only have a limited amount of data.