Robust Linear Regression
Also known as: robust regression, M-estimator regression, Huber regression, outlier-resistant regression
Robust linear regression fits a linear model between predictors and a continuous outcome while down-weighting or discarding influential outliers, preventing the few anomalous observations that OLS is famously sensitive to from distorting the entire estimated line. Major variants include Huber regression, iteratively reweighted least squares (IRLS), RANSAC, and Theil-Sen estimation.
Key highlights
- Coefficients remain stable and interpretable even when a non-trivial fraction of observations are outliers.
- Huber and bisquare losses give explicit control over the trade-off between efficiency and resistance.
- The down-weighting scheme naturally identifies influential observations as a by-product of fitting.
- IRLS retains computational similarity to OLS, keeping runtimes tractable for moderate datasets.
- Coefficient interpretation is identical to OLS — beta represents the expected change in outcome per unit change in predictor.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use robust linear regression when OLS assumptions are met except for the presence of outliers or heavy-tailed errors that you cannot confidently remove on subject-matter grounds, and when the relationship between predictors and outcome is plausibly linear in the bulk of the data. It is especially valuable in observational datasets — medical records, survey data, financial series — where extreme values reflect real-world variability rather than measurement error. Do not use it as a substitute for diagnosing and understanding outliers: if outliers encode genuine phenomena, they should be modelled, not suppressed. Also avoid it when sample sizes are very small (below ~30), because outlier identification itself becomes unreliable.
Strengths & limitations
- Coefficients remain stable and interpretable even when a non-trivial fraction of observations are outliers.
- Huber and bisquare losses give explicit control over the trade-off between efficiency and resistance.
- The down-weighting scheme naturally identifies influential observations as a by-product of fitting.
- IRLS retains computational similarity to OLS, keeping runtimes tractable for moderate datasets.
- Coefficient interpretation is identical to OLS — beta represents the expected change in outcome per unit change in predictor.
- Robust estimators are less efficient than OLS when errors truly are Gaussian — you pay a small precision cost in clean data.
- The delta (threshold) hyperparameter of the Huber loss requires tuning; a poor choice can either under- or over-suppress legitimate variance.
- Robust inference (sandwich standard errors or bootstrap) adds complexity compared to the standard OLS t-tests familiar to most reviewers.
- Extensions to high-dimensional settings (many predictors) are less mature than regularised OLS variants such as ridge or lasso.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How is robust linear regression different from simply removing outliers before OLS?
Outlier removal is a binary, subjective decision that permanently discards information. Robust regression down-weights influential points continuously and automatically, retains them in the dataset, and flags them for inspection — which is more transparent and reproducible.
Which variant should I choose: Huber, bisquare, RANSAC, or Theil-Sen?
Huber regression is the most common starting point and works well when outliers are moderate. Bisquare (Tukey) loss completely ignores extreme points and suits heavy contamination. RANSAC is suited to geometric or computer-vision data with a clear inlier structure. Theil-Sen is non-parametric and works well for small samples with moderate outliers.
Do I still need to check linearity and homoscedasticity?
Yes. Robust regression relaxes the normality-of-errors assumption and limits the damage from outliers, but it still assumes a linear relationship between predictors and outcome. Severe heteroscedasticity should be addressed separately, for example with weighted least squares or generalised linear models.
Can I use robust regression for classification?
No — robust linear regression is a regression method for a continuous outcome. For classification under outlier contamination, consider robust logistic regression or SVMs with soft-margin formulations.
How many outliers can robust regression tolerate?
The breakdown point — the fraction of contamination a method can withstand — depends on the estimator. Huber and bisquare M-estimators have a breakdown point of up to 50% in theory; LTS can achieve 50%. In practice, contamination above 20–30% warrants careful scrutiny before trusting any estimator.
Sources
- 1.Huber, P. J. (1964). Robust Estimation of a Location Parameter. Annals of Mathematical Statistics, 35(1), 73–101.
- 2.Rousseeuw, P. J. & Leroy, A. M. (1987). Robust Regression and Outlier Detection. Wiley.ISBN 978-0-471-85233-9
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Robust Linear Regression. ScholarGate. https://scholargate.app/machine-learning/robust-linear-regression