Bagging Ensemble
Also known as: bootstrap aggregating
Bagging, short for bootstrap aggregating, is an ensemble method that reduces variance by training multiple copies of a single learning algorithm on different random subsets of the training data. Each subset is created via bootstrap sampling—randomly drawing samples with replacement. Predictions are combined through majority voting (classification) or averaging (regression). Introduced by Leo Breiman in 1996, bagging forms the foundation for random forests and is particularly effective for reducing overfitting in high-variance models.
Key highlights
- Reduces variance without increasing bias, improving generalization.
- Works with any base learner type; can use unstable models like trees.
- Easily parallelizable—each bootstrap sample can be processed independently.
- Provides variance estimates through out-of-bag error calculation.
- Simple to implement with minimal hyperparameter tuning.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use bagging when working with high-variance models (e.g., decision trees, neural networks) that tend to overfit. It is most effective when base learners are unstable—sensitive to small changes in training data. Bagging is less effective for low-bias, low-variance models (e.g., k-NN with small k, linear regression on stable data). It is also well-suited for parallel computing since each base learner can be trained independently.
Strengths & limitations
- Reduces variance without increasing bias, improving generalization.
- Works with any base learner type; can use unstable models like trees.
- Easily parallelizable—each bootstrap sample can be processed independently.
- Provides variance estimates through out-of-bag error calculation.
- Simple to implement with minimal hyperparameter tuning.
- Provides no bias reduction; only effective for high-variance base learners.
- Computational cost scales linearly with the number of base learners.
- Bootstrap samples lack diversity compared to deliberate sampling strategies.
- May not improve performance when base models have high bias (e.g., linear models on complex data).
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How many bootstrap samples (B) should I use?
Start with B = 50-100 and monitor performance; typically, benefits plateau around B = 100-500. Increase B if variance reduction stalls. For critical applications, B > 1000 may be justified despite computational cost.
What is out-of-bag error and how do I use it?
Each bootstrap sample excludes approximately 1/3 of the original data (about 36.8% on average). These out-of-bag samples can be used to validate that specific model without needing a separate validation set. Average out-of-bag errors across all models estimate generalization error.
Is bagging better than cross-validation?
They serve different purposes. Cross-validation estimates model performance; bagging improves it. Use bagging to train an ensemble and out-of-bag estimates to validate, avoiding the overhead of k-fold cross-validation.
Can bagging improve low-variance models like k-NN?
Not effectively. Bagging is designed for high-variance models. Low-variance models (stable, simple) benefit little from variance reduction. Consider other ensemble methods or focus on bias reduction instead.
Sources
- 1.Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123-140.
- 2.Efron, B. (1979). Bootstrap methods: another look at the jackknife. The Annals of Statistics, 7(1), 1-26.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Bagging Ensemble. ScholarGate. https://scholargate.app/ensemble-learning/bagging-ensemble