Ensemble Linear Regression
Ensemble of Linear Regression Models (Bagged and Stacked Linear Regression) · Also known as: bagged linear regression, aggregated linear regression, stacked linear models, bootstrap-aggregated OLS
Ensemble Linear Regression combines multiple ordinary least-squares models — each fitted on a different bootstrap sample or feature subset — and averages their predictions. The technique, grounded in Breiman's bagging framework (1996), reduces variance and improves predictive stability compared with a single linear regression fit, while retaining the interpretability of linear assumptions.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Ensemble Linear Regression is appropriate when a linear relationship between predictors and a continuous outcome is expected or required by domain convention, yet a single OLS fit shows high variance across repeated samples or cross-validation folds. It is particularly useful when the dataset is moderately small (roughly 30–500 observations) where ordinary regression coefficients are unstable. Do not use it when the relationship is clearly nonlinear and a tree-based ensemble would fit better, or when full coefficient interpretability (exact p-values, standard errors) is mandatory for reporting — averaged coefficients do not carry standard inferential machinery. Also avoid it when the training set is large enough that a single well-regularized regression (Ridge, Lasso) already yields stable estimates.
Strengths & limitations
- Reduces prediction variance compared with a single OLS fit without requiring a new model family.
- Averaged coefficients still convey direction of effect, preserving a degree of interpretability.
- Out-of-bag predictions provide an internal validation estimate at no extra data cost.
- Straightforward to implement using any standard linear regression library inside a bootstrap loop.
- Compatible with regularization (Ridge, Lasso) at the base-model level for further stability.
- Does not reduce bias: if a single linear regression is already biased (wrong functional form), the ensemble inherits that bias.
- Loses the clean inferential machinery of OLS — p-values and confidence intervals from individual base models are not directly interpretable.
- Computationally more expensive than a single fit, though usually trivial compared with tree-based ensembles.
- Provides no benefit over well-tuned Ridge or Lasso regression when the dataset is large and collinearity is the main problem.
Frequently asked
How many bootstrap models should I use?
Start with 100 to 200 base models. Increase until the out-of-bag error stabilises. Diminishing returns usually appear well before 500 models.
Is this the same as Ridge regression?
No. Ridge regression adds an L2 penalty to a single fit to handle collinearity. Ensemble linear regression averages many independently fitted models to reduce variance from sampling instability. They are complementary: each base model in the ensemble can itself be a Ridge regression.
Can I still interpret the coefficients?
You can compute the mean and standard deviation of each coefficient across B bootstrap fits. The mean coefficient indicates direction of effect; the standard deviation reflects sampling variability. However, formal p-values from individual OLS fits do not apply to these averaged estimates.
When should I prefer a random forest over this method?
When the relationship between predictors and the outcome is nonlinear or involves complex interactions, a random forest (which grows nonlinear trees) will capture those patterns far better. Ensemble linear regression is best when domain knowledge or reporting requirements call for a linear model.
Does averaging predictions reduce bias?
No. Averaging reduces variance but leaves bias unchanged. If every base linear model is biased — for example because the true relationship is quadratic — the ensemble average will be equally biased.
Sources
- Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140. DOI: 10.1007/BF00058655 ↗
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed., Ch. 8). Springer. ISBN: 978-0-387-84857-0
How to cite this page
ScholarGate. (2026, June 3). Ensemble of Linear Regression Models (Bagged and Stacked Linear Regression). ScholarGate. https://scholargate.app/en/machine-learning/ensemble-linear-regression
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- BaggingMachine learning↔ compare
- Linear Regression (ML)Machine learning↔ compare
- Random ForestMachine learning↔ compare
- Regularized linear regressionMachine learning↔ compare
- Ridge RegressionMachine learning↔ compare
- Voting EnsembleMachine learning↔ compare