Ensemble Logistic Regression
Also known as: logistic regression ensemble, bagged logistic regression, aggregated logistic regression, logistic ensemble classifier
Ensemble Logistic Regression trains multiple logistic regression classifiers on varied subsets or perturbations of the training data and combines their probability estimates by averaging or voting. The approach preserves logistic regression's probabilistic interpretability while reducing variance and improving predictive stability through aggregation.
Key highlights
- Reduces variance of logistic regression estimates without sacrificing probability calibration.
- Maintains the probabilistic output and interpretability of logistic regression at the individual model level.
- More stable coefficient estimates under data perturbation compared to a single logistic regression.
- Simple to implement using standard bagging frameworks applied to a logistic base estimator.
- Out-of-bag samples from bagging provide an internal performance estimate without extra hold-out data.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use Ensemble Logistic Regression when logistic regression is the preferred model for its interpretability and probability calibration, but variance reduction is needed due to noisy or small-to-medium-sized datasets. It is well suited for binary and multiclass classification on tabular data where you want stable, calibrated probabilities and some resilience to outliers. Do not use it as a substitute for addressing class imbalance, data leakage, or severe underfitting — these require separate remedies. If very high predictive accuracy is the sole goal, gradient boosting or random forest often outperform it.
Strengths & limitations
- Reduces variance of logistic regression estimates without sacrificing probability calibration.
- Maintains the probabilistic output and interpretability of logistic regression at the individual model level.
- More stable coefficient estimates under data perturbation compared to a single logistic regression.
- Simple to implement using standard bagging frameworks applied to a logistic base estimator.
- Out-of-bag samples from bagging provide an internal performance estimate without extra hold-out data.
- Individual ensemble members no longer yield a single interpretable coefficient vector, reducing explanatory transparency.
- Does not address the bias of logistic regression — if the decision boundary is nonlinear, bagging many logistic regressors still yields a linear ensemble.
- Computationally more expensive than a single logistic regression by a factor of B (number of members).
- Performance gains over a well-tuned single logistic regression are modest when data are large and clean.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How many base logistic regression models should the ensemble contain?
Typically 50 to 200 members. Variance reduction diminishes past a certain point, so evaluate out-of-bag performance as the count increases and stop when gains plateau.
Does averaging probabilities differ from a majority vote?
Yes. Averaging probabilities retains more information — especially the confidence near the decision boundary — and generally produces better-calibrated predictions than majority voting on binary outputs.
Can ensemble logistic regression handle nonlinear decision boundaries?
No. Each member is a linear classifier, so the ensemble remains linear in the original feature space. For nonlinear boundaries, apply feature engineering or switch to a nonlinear model such as random forest or gradient boosting.
How does ensemble logistic regression compare to regularized logistic regression?
Regularization (L1/L2) reduces variance through shrinkage of a single model; bagging reduces variance by averaging multiple models. The two approaches can be combined: regularize each base classifier and then aggregate them.
Is the ensemble interpretable?
Individual members are interpretable logistic regressions, but the ensemble as a whole does not expose a single coefficient vector. For post-hoc explanations across the ensemble, use averaged SHAP values or average the standardized coefficients with caution.
Sources
- 1.Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.
- 2.Polikar, R. (2006). Ensemble based systems in decision making. IEEE Circuits and Systems Magazine, 6(3), 21–45.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Ensemble Logistic Regression. ScholarGate. https://scholargate.app/machine-learning/ensemble-logistic-regression