Online Linear Regression
Online Linear Regression (Incremental Least-Squares) · Also known as: incremental linear regression, streaming linear regression, recursive least squares regression, stochastic gradient descent regression
Online Linear Regression fits a linear model one observation at a time, updating weights incrementally as each new data point arrives. Unlike batch least-squares, it never needs to store or re-process the full dataset, making it the natural choice for streaming data, very large datasets, and environments where the data-generating process can shift over time.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use online linear regression when data arrives as a continuous stream and predictions must be served immediately, when the dataset is too large to hold in memory for a batch fit, or when the relationship between predictors and outcome may drift over time and the model needs to adapt. It is also a good choice as a lightweight baseline in real-time production systems. Avoid it when the relationship is strongly nonlinear, when feature engineering must be done holistically over the full dataset, or when you need exact OLS coefficients with confidence intervals for statistical inference — use batch linear regression in those cases.
Strengths & limitations
- Processes data one observation at a time with O(p) memory, where p is the number of features — no full dataset required.
- Adapts continuously to concept drift, making it suitable for non-stationary environments such as financial time series or sensor streams.
- Computationally cheap per update: a single dot product and a weight vector addition.
- Naturally parallelisable across multiple streams or shards.
- Produces predictions in real time without waiting for a batch re-fit.
- Sensitive to the choice of learning rate: too large causes oscillation, too small causes slow convergence.
- Provides no closed-form standard errors or p-values, so classical statistical inference is not directly available.
- Can struggle with highly correlated features (multicollinearity) unless L2 regularisation is added.
- Performance degrades relative to batch OLS when all data is already available and there is no concept drift.
Frequently asked
How does online linear regression differ from standard (batch) linear regression?
Batch linear regression solves the normal equations on the entire dataset at once, giving the exact OLS solution. Online linear regression updates weights incrementally after each observation using a gradient step, converging to a near-OLS solution without ever loading all data — at the cost of losing exact inference statistics.
What learning rate should I use?
A common starting point is eta = 0.01 divided by the expected squared norm of the feature vectors. Grid-search or adaptive methods (AdaGrad, Adam) automatically scale the learning rate per feature and are often easier to tune than a fixed global rate.
Can online linear regression handle concept drift?
Yes. A constant (non-decaying) learning rate keeps the model responsive to drift. For abrupt shifts, explicit drift-detection tests (e.g., ADWIN) can trigger a partial weight reset. A decaying rate is better when the relationship is stationary and you want eventual convergence.
Is regularisation possible in the online setting?
Yes. Adding an L2 penalty gives the online Ridge (Tikhonov) update: subtract a small fraction of the current weight vector at each step. This prevents weights from growing large on sparse or noisy streams and is strongly recommended in high-dimensional settings.
How do I evaluate an online model fairly?
Use prequential (test-then-train) evaluation: record the prediction error on each observation before using it to update the model. This gives an unbiased, sequential estimate of generalisation error without a separate held-out test set.
Sources
- Shalev-Shwartz, S. (2012). Online Learning and Online Convex Optimization. Foundations and Trends in Machine Learning, 4(2), 107–194. DOI: 10.1561/2200000018 ↗
- Haykin, S. (2002). Adaptive Filter Theory (4th ed.). Prentice Hall. ISBN: 978-0130901262
How to cite this page
ScholarGate. (2026, June 3). Online Linear Regression (Incremental Least-Squares). ScholarGate. https://scholargate.app/en/machine-learning/online-linear-regression
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Linear Regression (ML)Machine learning↔ compare
- Online LearningMachine learning↔ compare
- Online Logistic RegressionMachine learning↔ compare
- Regularized linear regressionMachine learning↔ compare
- Ridge RegressionMachine learning↔ compare
- Stochastic Gradient DescentMachine learning↔ compare