Notes
🔍

Stacking

When one model isn’t quite reliable enough, a natural idea is to ask several models and combine their opinions. Stacking is a disciplined way to do that: it learns how to combine models rather than hard-coding the combination rule.

What stacking is

Stacking (short for “stacked generalization”) is an ensemble method where you train multiple base models (level-0 models) and then train a meta-model (level-1 model) to combine their predictions. Instead of averaging or voting, the meta-model learns patterns like “trust model A more for these cases, but model B for those.” The meta-model’s inputs are typically the base models’ predicted probabilities (classification) or predicted values (regression).

How it’s trained without cheating

The key technical detail is avoiding data leakage. If the meta-model sees base-model predictions made on the same data those base models were trained on, it gets unrealistically easy signals and won’t generalize. Stacking uses out-of-fold predictions:

  • Split training data into K folds.
  • For each fold: train each base model on K−1 folds, predict on the held-out fold.
  • Concatenate these held-out predictions to form the meta-model training set.
  • Retrain base models on all training data, then use them + the meta-model at test time.

Why it matters (and where you see it)

Stacking can beat any single model when base models make different kinds of mistakes—e.g., combining LogisticRegression, RandomForest, and GradientBoosting for fraud detection, or mixing linear and tree models for house-price prediction. In scikit-learn, this is implemented as StackingClassifier and StackingRegressor, with cross-validation built in to generate out-of-fold meta-features.

Stacking is an ensemble method that trains multiple base models and then fits a meta-learner to combine their predictions, typically using out-of-fold predictions to avoid training on its own targets. It matters because it can exploit complementary strengths of diverse models to improve generalization beyond any single learner or simple averaging; without careful validation, it can overfit and give misleading performance estimates.

Think of picking a restaurant with friends. One friend checks reviews, another cares about price, another looks at distance. Instead of letting one person decide, you ask a final “judge” friend to listen to everyone and make the best call. Stacking in machine learning is like that. You train several different models to make predictions (like “spam or not” or “house price”), then train one more model to combine those predictions into a final answer. It matters because different models have different strengths, and stacking can turn a set of decent predictors into a stronger, more reliable one.