Stacked Generalization

When you train a few different models on the same problem, they’ll make different kinds of mistakes. Stacked generalization (usually just called stacking) is a way to turn those differences into an advantage by learning how to combine models, instead of picking one or averaging them blindly.

How stacking works

Stacking is an ensemble method with two layers:

Base models (level-0 models): diverse predictors such as a random forest, gradient boosting, and logistic regression.
A meta-model (level-1 model): a model trained to combine the base models’ predictions into a final prediction.

The key detail is how you generate training data for the meta-model. You use out-of-fold predictions from cross-validation: for each fold, train base models on the other folds and predict the held-out fold. Those held-out predictions become features for the meta-model, preventing it from “cheating” by seeing base-model predictions made on data they were trained on.

Why it matters (and what can go wrong)

Stacking matters because it can learn smarter combinations than simple voting/averaging—e.g., “trust model A more for high-income applicants, model B more for sparse text.” If you skip out-of-fold predictions and train the meta-model on in-sample base predictions, you get data leakage and overly optimistic validation scores that collapse in production.

Practical examples and tools

In spam detection, you might stack a linear text model with a tree model using metadata (sender reputation, links). In credit scoring, stacking can blend a calibrated LogisticRegression with XGBoost to improve ranking and probability quality. In scikit-learn, this is implemented as:

from sklearn.ensemble import StackingClassifier

Stacked Generalization (or stacking) is an ensemble technique that trains multiple base models and then fits a meta-learner on their out-of-sample predictions to produce the final prediction. The meta-learner learns how to weight or correct base models based on their strengths and errors. It matters because it can outperform any single model or simple averaging by exploiting complementary inductive biases while controlling overfitting via proper cross-validation.

Think of getting medical advice: you might ask a family doctor, a specialist, and a pharmacist. Each gives a useful opinion, but you may trust a final “coordinator” who listens to all of them and makes the best overall call.

Stacked Generalization (often called stacking) is that idea in machine learning. Instead of relying on one model, you train several different models and let each one make its own prediction. Then you train one more model that learns how to combine those predictions into a single, better answer. This often improves accuracy because different models make different kinds of mistakes, and stacking can learn which ones to trust in which situations.