Notes
🔍

Bootstrapping

When you only have one dataset, it’s natural to wonder: “How stable is my model’s performance estimate?” Bootstrapping answers that by creating many “new” datasets from the one you already have—without collecting any new data.

What bootstrapping is doing

In model evaluation, bootstrapping is a resampling method: you repeatedly sample with replacement from your original dataset to form many bootstrap datasets (each the same size as the original). Because sampling is with replacement, some rows appear multiple times and some are left out. You train the model on each bootstrap sample and evaluate it, typically on the points not selected for that sample (the out-of-bag set) or by using a bootstrap-specific correction (like the .632 bootstrap).

Why it matters for supervised learning

Bootstrapping gives you a distribution of performance estimates, not just a single number. That unlocks practical things you care about:

  • Uncertainty: build confidence intervals for accuracy, AUC, RMSE, etc.
  • Stability: detect when results swing wildly depending on which examples you happened to get.
  • Fair comparisons: compare two models using paired bootstrap differences in metrics.
Concrete example

Suppose you’re building a credit default classifier with 5,000 labeled loans. You can bootstrap 1,000 times, fit (say) scikit-learn’s LogisticRegression each time, and compute AUC on the out-of-bag loans. If the AUC ranges from 0.71 to 0.83 across resamples, that tells you the “true” generalization performance is uncertain—valuable information before shipping the model.

Bootstrapping is a resampling-based evaluation method that repeatedly draws samples of size n from the training set with replacement, fits the model on each resample, and evaluates it on the out-of-bag (unsampled) points to estimate performance and uncertainty (e.g., confidence intervals). It matters because it provides a data-efficient way to quantify generalization error and variability when a single train/test split is unstable or data is limited.

Imagine you want to know how reliable a restaurant’s rating is, but you only have a small stack of reviews. One trick is to make lots of “new” stacks by randomly picking reviews from the original pile, allowing repeats. You then see how much the rating changes across these stacks.

That’s the idea behind bootstrapping in machine learning. It’s a way to test how steady a model’s performance is when data is limited. By repeatedly reusing the same dataset in slightly different mixes, bootstrapping helps estimate how much your accuracy (or error) might vary in the real world.