Logistic Regression (ML)
Also known as: logit model, logit regression, binomial logistic regression, maximum entropy classifier
Logistic regression is a foundational probabilistic classifier that models the log-odds of a binary (or multinomial) outcome as a linear function of the predictors. Introduced by D. R. Cox in 1958, it remains one of the most widely used and interpretable classification methods in both statistics and machine learning, valued for its calibrated probability outputs and clear coefficient interpretation.
Key highlights
- Produces calibrated class probabilities, not just labels, which are directly usable for ranking and decision thresholding.
- Coefficients are interpretable as log-odds ratios, meeting the standards expected in clinical, social-science, and policy research.
- Computationally fast and memory-efficient even on large datasets.
- Regularised variants (Ridge, Lasso, Elastic Net) handle high-dimensional features and perform automatic feature selection.
- Assumption-light relative to discriminant analysis: does not require multivariate normality of predictors.
- Robust and well-understood; serves as a strong, credible baseline against which complex models should be compared.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use logistic regression when the outcome is binary or categorical, the sample size is at least moderate (n > 10 events per predictor as a rule of thumb), and interpretable coefficients or calibrated probabilities are required. It is the default baseline classifier in social science, medicine, and epidemiology and should be tried before more complex models. Avoid when the decision boundary is strongly nonlinear, when many interaction terms would be needed, or when prediction accuracy on complex tabular data matters more than interpretability — in those cases tree-based ensembles or SVMs typically outperform it.
Strengths & limitations
- Produces calibrated class probabilities, not just labels, which are directly usable for ranking and decision thresholding.
- Coefficients are interpretable as log-odds ratios, meeting the standards expected in clinical, social-science, and policy research.
- Computationally fast and memory-efficient even on large datasets.
- Regularised variants (Ridge, Lasso, Elastic Net) handle high-dimensional features and perform automatic feature selection.
- Assumption-light relative to discriminant analysis: does not require multivariate normality of predictors.
- Robust and well-understood; serves as a strong, credible baseline against which complex models should be compared.
- Assumes a linear relationship between features and the log-odds of the outcome; nonlinear patterns require manual feature engineering or a nonlinear model.
- Sensitive to multicollinearity among predictors: correlated features inflate coefficient standard errors and destabilise estimates.
- Requires a reasonable number of events per predictor (rule of thumb: at least 10); sparse outcome classes cause convergence problems and overfitting.
- May underperform ensemble methods (random forest, gradient boosting) on complex tabular data with many interactions.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
When should I prefer logistic regression over a random forest?
When you need calibrated probabilities, interpretable coefficients, or must satisfy reporting standards that require explicit odds ratios — as in clinical trials or policy evaluation. Random forest typically achieves higher predictive accuracy but provides no direct coefficient interpretation.
How many observations do I need?
A widely cited rule of thumb is at least 10 events (positive outcomes) per predictor variable. With fewer events, parameter estimates become unstable and confidence intervals unreliable. Regularisation or penalised likelihood can extend the usable range somewhat.
Should I standardise my features?
Standardising features is not required for prediction, but it is important when comparing coefficient magnitudes across predictors or when using regularisation (L1 or L2), because penalties treat all coefficients on the same scale.
What is the difference between binary and multinomial logistic regression?
Binary logistic regression models a two-class outcome using the sigmoid function. Multinomial (softmax) logistic regression generalises this to three or more unordered classes, estimating one set of coefficients per class relative to a reference class.
How do I handle class imbalance?
Set class_weight='balanced' in scikit-learn to re-weight the loss function, or oversample the minority class (SMOTE). Always evaluate with AUC-ROC and precision-recall curves rather than accuracy, and tune the classification threshold on a validation set.
Sources
- 1.Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society, Series B, 20(2), 215–242.
- 2.James, G., Witten, D., Hastie, T. & Tibshirani, R. (2013). An Introduction to Statistical Learning (Ch. 4). Springer.ISBN 978-1-4614-7138-7
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Logistic regression (ML). ScholarGate. https://scholargate.app/machine-learning/logistic-regression-ml