Bagging (Bootstrap Aggregating)
Also known as: Bootstrap Aggregating, bootstrap aggregation, bagged ensemble, bagged predictor, variance reduction ensemble
Bagging, short for Bootstrap Aggregating, is an ensemble meta-algorithm introduced by Leo Breiman in 1996 that trains multiple copies of a base learner on independently drawn bootstrap samples of the training data and combines their predictions — by averaging for regression or majority vote for classification — to produce a final predictor with substantially lower variance than any single base learner.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+16 more
When to use it
Bagging is appropriate whenever the base learning algorithm is high-variance — that is, sensitive to small perturbations in training data — and reducing that variance is the main source of error. Fully grown, unpruned decision trees are the canonical example. Bagging is suitable for both classification and regression on tabular data, requires no assumptions about data distribution, and handles mixed feature types. It is most beneficial when the dataset is large enough to support B independent bootstrap samples of quality (at least about 30 observations is a practical lower bound, though hundreds are preferable). Bagging is not beneficial when the base learner is already stable (such as linear regression on well-specified problems) or when the dominant error source is bias rather than variance — in such cases boosting methods are more appropriate.
Strengths & limitations
- Directly reduces estimator variance without increasing bias, producing more stable predictions than any single base model.
- Trivially parallelisable: each bootstrap iteration is independent, making large ensembles computationally tractable.
- Built-in OOB error estimation provides an honest generalisation estimate at no extra data cost.
- Model-agnostic meta-algorithm: compatible with any sufficiently unstable base learner (trees, neural networks, nearest neighbours).
- Robust to outliers in the training set: any single unusual observation is absent from most bootstrap samples.
- Does not reduce bias: if the base learner underfits, bagging cannot correct it.
- Ensemble predictions are harder to interpret than a single model; no analogue of a single decision tree's visual structure.
- Memory and storage scale linearly with B: keeping hundreds of large models in memory can be prohibitive.
- Gains diminish rapidly as B grows; performance usually plateaus well before B = 500.
- On stable base learners (e.g. linear regression with correct specification) bagging yields negligible improvement.
Frequently asked
How many bootstrap samples B should I use?
Breiman (1996) found that performance stabilises relatively quickly; B = 50 to 100 is often sufficient for decision trees. Increasing B beyond a few hundred yields diminishing returns. Monitor the OOB error as B grows and stop when it plateaus.
What is the difference between bagging and Random Forest?
Random Forest extends bagging by adding random feature subsampling at each split of every tree. This additional randomisation further de-correlates the base learners, reducing variance beyond what bagging alone achieves. Bagging with fully grown trees is therefore a special case of Random Forest where all features are considered at every split.
Can bagging be applied to learners other than decision trees?
Yes. Any learner that is unstable — whose predictions change substantially when the training data change slightly — can benefit from bagging. Neural networks, k-nearest neighbours, and unpruned regression trees all qualify. Stable learners such as linear regression with a well-specified model benefit little because there is little variance to reduce.
Is the OOB error a reliable substitute for cross-validation?
For moderate to large datasets the OOB error is a nearly unbiased estimate of generalisation error and is computationally free since no extra model fits are required. On very small datasets (n below about 50) OOB estimates can be optimistic and cross-validation is preferable.
Sources
- Breiman, L. (1996). Bagging Predictors. Machine Learning, 24(2), 123–140. DOI: 10.1007/BF00058655 ↗
- Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed., Ch. 8.7). Springer. ISBN: 978-0-387-84857-0
- James, G., Witten, D., Hastie, T. & Tibshirani, R. (2013). An Introduction to Statistical Learning (Ch. 8.2). Springer. ISBN: 978-1-4614-7138-7
How to cite this page
ScholarGate. (2026, June 3). Bagging (Bootstrap Aggregating). ScholarGate. https://scholargate.app/en/machine-learning/bagging
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- AdaBoostMachine learning↔ compare
- Decision TreeMachine learning↔ compare
- Gradient BoostingMachine learning↔ compare
- Random ForestMachine learning↔ compare
- XGBoostMachine learning↔ compare