Linear Regression (ML)
Also known as: ordinary least squares regression, OLS, least squares regression, multiple linear regression
Linear regression fits a straight-line relationship between one or more input features and a continuous numeric outcome by minimising the sum of squared prediction errors. As a machine-learning model it is trained on labeled examples and evaluated on held-out data, making it the simplest supervised learning baseline for any regression task.
Key highlights
- Coefficients are directly interpretable: each value is a marginal effect in the outcome's units.
- Computationally trivial — fits almost instantly even on large datasets.
- Provides standard errors and p-values, supporting formal hypothesis testing.
- Serves as the indispensable baseline against which more complex models should be compared.
- Works well and is fully justified when the linearity assumption holds.
- Highly compatible with regularisation extensions (Ridge, Lasso, Elastic Net) when predictors are many.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use linear regression when the outcome is continuous, the expected relationship between predictors and outcome is approximately linear, and interpretable coefficients are needed. It is the right starting baseline before trying more complex models, and a good final model when its assumptions hold. Do not use it for binary or categorical outcomes (use logistic regression instead), when strong nonlinear patterns or interactions dominate, or when the number of predictors exceeds the sample size without regularisation.
Strengths & limitations
- Coefficients are directly interpretable: each value is a marginal effect in the outcome's units.
- Computationally trivial — fits almost instantly even on large datasets.
- Provides standard errors and p-values, supporting formal hypothesis testing.
- Serves as the indispensable baseline against which more complex models should be compared.
- Works well and is fully justified when the linearity assumption holds.
- Highly compatible with regularisation extensions (Ridge, Lasso, Elastic Net) when predictors are many.
- Assumes a linear relationship; misses curves, thresholds, and interactions unless they are manually encoded.
- Sensitive to outliers because squaring errors amplifies the influence of extreme points.
- Multicollinearity among predictors inflates standard errors and destabilises coefficient estimates.
- Requires that residuals be approximately normally distributed and homoscedastic for valid inference.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How is linear regression different from logistic regression?
Linear regression predicts a continuous outcome directly. Logistic regression models the log-odds of a binary outcome and outputs probabilities bounded between 0 and 1. Using linear regression for a binary outcome can produce predictions outside [0, 1] and violates the constant-variance assumption.
Do I need to standardise my features?
Standardisation is not required for predictions, but it makes coefficients directly comparable when features have very different scales. It is especially recommended when using regularised variants such as Ridge or Lasso.
How do I handle many predictors relative to sample size?
Switch to regularised linear regression (Ridge for shrinkage, Lasso for sparsity, Elastic Net for both). OLS becomes unreliable or undefined when the number of predictors approaches or exceeds the number of observations.
What residual plots should I inspect?
At minimum: (1) residuals vs. fitted values to check for patterns suggesting nonlinearity or heteroscedasticity, and (2) a Q-Q plot of residuals to assess normality. A scale-location plot and leverage/influence statistics (Cook's distance) are also informative.
Is a high R-squared enough to declare a good model?
No. R-squared measures in-sample fit and can be inflated by irrelevant predictors. Always report cross-validated or test-set RMSE alongside R-squared, and inspect residual diagnostics to confirm the model's assumptions are satisfied.
Sources
- 1.Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed., Ch. 3). Springer.ISBN 978-0-387-84858-7
- 2.James, G., Witten, D., Hastie, T. & Tibshirani, R. (2013). An Introduction to Statistical Learning (Ch. 3). Springer.ISBN 978-1-4614-7138-7
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Linear Regression (ML). ScholarGate. https://scholargate.app/machine-learning/linear-regression-ml