Stacked Generalization
Also known as: stacking, meta-learning
Stacked generalization, or stacking, is a two-level ensemble method where base-level classifiers are trained on the original data, and a meta-learner is trained on the predictions of the base classifiers. The meta-learner learns how to best combine base predictions rather than using fixed aggregation rules. Introduced by David Wolpert in 1992, stacking achieves state-of-the-art performance by automatically learning the optimal weighting and interaction patterns among base models.
Key highlights
- Learns optimal aggregation weights automatically, outperforming fixed voting rules.
- Adapts to the strengths and weaknesses of individual base classifiers.
- Handles heterogeneous base learners (different algorithms, hyperparameters).
- Often achieves state-of-the-art performance in machine learning competitions.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use stacking when you have diverse base classifiers with moderate individual accuracy and want to optimize their combination. It is especially effective when base models have different strengths and weaknesses (e.g., decision trees, neural networks, SVMs). Avoid stacking if base classifiers are highly correlated or if computational cost is critical, as stacking requires training an additional meta-model.
Strengths & limitations
- Learns optimal aggregation weights automatically, outperforming fixed voting rules.
- Adapts to the strengths and weaknesses of individual base classifiers.
- Handles heterogeneous base learners (different algorithms, hyperparameters).
- Often achieves state-of-the-art performance in machine learning competitions.
- Requires careful design to avoid overfitting: meta-learner can memorize base predictions if not properly validated.
- Increased computational cost: training K+1 models instead of just K.
- Meta-learner selection is often problem-specific; no universal best choice.
- Interpretability decreases as complexity increases—difficult to understand final decision-making logic.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How do I avoid overfitting in stacking?
Use k-fold cross-validation to generate meta-features: each fold's test predictions become meta-training data. This prevents the meta-learner from seeing predictions on data used to train base classifiers. Always use a separate hold-out validation set for final evaluation.
What model should I use as the meta-learner?
Simple models often work best: logistic regression, linear regression, or a small decision tree. Complex meta-learners (random forests, neural networks) risk overfitting to the meta-features. Start simple and only add complexity if validation performance plateaus.
Should base classifiers be trained on the same data?
No. Diverse base classifiers trained on different data samples (bagging) or with different algorithms improve meta-learner generalization. Identical base classifiers produce identical predictions, offering no useful information to the meta-learner.
Can I use stacking with regression and classification?
Yes. For regression, the meta-learner predicts continuous values from base regressor outputs. For classification, it predicts classes from base classifier probability scores. The principle is identical.
Sources
- 1.Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241-259.
- 2.Breiman, L. (1996). Stacked regressions. Machine Learning, 24(1), 49-64.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Stacked Generalization. ScholarGate. https://scholargate.app/ensemble-learning/stacked-generalization