Robust Linear Regression
Robust Linear Regression (Outlier-Resistant Estimation) · Also known as: robust regression, M-estimator regression, Huber regression, outlier-resistant regression
Robust linear regression fits a linear model between predictors and a continuous outcome while down-weighting or discarding influential outliers, preventing the few anomalous observations that OLS is famously sensitive to from distorting the entire estimated line. Major variants include Huber regression, iteratively reweighted least squares (IRLS), RANSAC, and Theil-Sen estimation.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+1 more
When to use it
Use robust linear regression when OLS assumptions are met except for the presence of outliers or heavy-tailed errors that you cannot confidently remove on subject-matter grounds, and when the relationship between predictors and outcome is plausibly linear in the bulk of the data. It is especially valuable in observational datasets — medical records, survey data, financial series — where extreme values reflect real-world variability rather than measurement error. Do not use it as a substitute for diagnosing and understanding outliers: if outliers encode genuine phenomena, they should be modelled, not suppressed. Also avoid it when sample sizes are very small (below ~30), because outlier identification itself becomes unreliable.
Strengths & limitations
- Coefficients remain stable and interpretable even when a non-trivial fraction of observations are outliers.
- Huber and bisquare losses give explicit control over the trade-off between efficiency and resistance.
- The down-weighting scheme naturally identifies influential observations as a by-product of fitting.
- IRLS retains computational similarity to OLS, keeping runtimes tractable for moderate datasets.
- Coefficient interpretation is identical to OLS — beta represents the expected change in outcome per unit change in predictor.
- Robust estimators are less efficient than OLS when errors truly are Gaussian — you pay a small precision cost in clean data.
- The delta (threshold) hyperparameter of the Huber loss requires tuning; a poor choice can either under- or over-suppress legitimate variance.
- Robust inference (sandwich standard errors or bootstrap) adds complexity compared to the standard OLS t-tests familiar to most reviewers.
- Extensions to high-dimensional settings (many predictors) are less mature than regularised OLS variants such as ridge or lasso.
Frequently asked
How is robust linear regression different from simply removing outliers before OLS?
Outlier removal is a binary, subjective decision that permanently discards information. Robust regression down-weights influential points continuously and automatically, retains them in the dataset, and flags them for inspection — which is more transparent and reproducible.
Which variant should I choose: Huber, bisquare, RANSAC, or Theil-Sen?
Huber regression is the most common starting point and works well when outliers are moderate. Bisquare (Tukey) loss completely ignores extreme points and suits heavy contamination. RANSAC is suited to geometric or computer-vision data with a clear inlier structure. Theil-Sen is non-parametric and works well for small samples with moderate outliers.
Do I still need to check linearity and homoscedasticity?
Yes. Robust regression relaxes the normality-of-errors assumption and limits the damage from outliers, but it still assumes a linear relationship between predictors and outcome. Severe heteroscedasticity should be addressed separately, for example with weighted least squares or generalised linear models.
Can I use robust regression for classification?
No — robust linear regression is a regression method for a continuous outcome. For classification under outlier contamination, consider robust logistic regression or SVMs with soft-margin formulations.
How many outliers can robust regression tolerate?
The breakdown point — the fraction of contamination a method can withstand — depends on the estimator. Huber and bisquare M-estimators have a breakdown point of up to 50% in theory; LTS can achieve 50%. In practice, contamination above 20–30% warrants careful scrutiny before trusting any estimator.
Sources
- Huber, P. J. (1964). Robust Estimation of a Location Parameter. Annals of Mathematical Statistics, 35(1), 73–101. DOI: 10.1214/aoms/1177703732 ↗
- Rousseeuw, P. J. & Leroy, A. M. (1987). Robust Regression and Outlier Detection. Wiley. ISBN: 978-0-471-85233-9
How to cite this page
ScholarGate. (2026, June 3). Robust Linear Regression (Outlier-Resistant Estimation). ScholarGate. https://scholargate.app/en/machine-learning/robust-linear-regression
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Huber RegressionStatistics↔ compare
- Lasso RegressionMachine learning↔ compare
- Linear Regression (ML)Machine learning↔ compare
- Quantile RegressionEconometrics↔ compare
- Regularized linear regressionMachine learning↔ compare