Ensemble Logistic Regression
Ensemble Logistic Regression (Combined Logistic Classifier Ensemble) · Also known as: logistic regression ensemble, bagged logistic regression, aggregated logistic regression, logistic ensemble classifier
Ensemble Logistic Regression trains multiple logistic regression classifiers on varied subsets or perturbations of the training data and combines their probability estimates by averaging or voting. The approach preserves logistic regression's probabilistic interpretability while reducing variance and improving predictive stability through aggregation.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use Ensemble Logistic Regression when logistic regression is the preferred model for its interpretability and probability calibration, but variance reduction is needed due to noisy or small-to-medium-sized datasets. It is well suited for binary and multiclass classification on tabular data where you want stable, calibrated probabilities and some resilience to outliers. Do not use it as a substitute for addressing class imbalance, data leakage, or severe underfitting — these require separate remedies. If very high predictive accuracy is the sole goal, gradient boosting or random forest often outperform it.
Strengths & limitations
- Reduces variance of logistic regression estimates without sacrificing probability calibration.
- Maintains the probabilistic output and interpretability of logistic regression at the individual model level.
- More stable coefficient estimates under data perturbation compared to a single logistic regression.
- Simple to implement using standard bagging frameworks applied to a logistic base estimator.
- Out-of-bag samples from bagging provide an internal performance estimate without extra hold-out data.
- Individual ensemble members no longer yield a single interpretable coefficient vector, reducing explanatory transparency.
- Does not address the bias of logistic regression — if the decision boundary is nonlinear, bagging many logistic regressors still yields a linear ensemble.
- Computationally more expensive than a single logistic regression by a factor of B (number of members).
- Performance gains over a well-tuned single logistic regression are modest when data are large and clean.
Frequently asked
How many base logistic regression models should the ensemble contain?
Typically 50 to 200 members. Variance reduction diminishes past a certain point, so evaluate out-of-bag performance as the count increases and stop when gains plateau.
Does averaging probabilities differ from a majority vote?
Yes. Averaging probabilities retains more information — especially the confidence near the decision boundary — and generally produces better-calibrated predictions than majority voting on binary outputs.
Can ensemble logistic regression handle nonlinear decision boundaries?
No. Each member is a linear classifier, so the ensemble remains linear in the original feature space. For nonlinear boundaries, apply feature engineering or switch to a nonlinear model such as random forest or gradient boosting.
How does ensemble logistic regression compare to regularized logistic regression?
Regularization (L1/L2) reduces variance through shrinkage of a single model; bagging reduces variance by averaging multiple models. The two approaches can be combined: regularize each base classifier and then aggregate them.
Is the ensemble interpretable?
Individual members are interpretable logistic regressions, but the ensemble as a whole does not expose a single coefficient vector. For post-hoc explanations across the ensemble, use averaged SHAP values or average the standardized coefficients with caution.
Sources
- Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140. DOI: 10.1007/BF00058655 ↗
- Polikar, R. (2006). Ensemble based systems in decision making. IEEE Circuits and Systems Magazine, 6(3), 21–45. DOI: 10.1109/MCAS.2006.1688199 ↗
How to cite this page
ScholarGate. (2026, June 3). Ensemble Logistic Regression (Combined Logistic Classifier Ensemble). ScholarGate. https://scholargate.app/en/machine-learning/ensemble-logistic-regression
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- BoostingMachine learning↔ compare
- Logistic regression (ML)Machine learning↔ compare
- Random ForestMachine learning↔ compare
- Semi-supervised Logistic RegressionMachine learning↔ compare
- StackingMachine learning↔ compare
- Voting EnsembleMachine learning↔ compare