Why are the normal equations discouraged for least squares?

Forming the normal equations squares the condition number of the problem, so rounding error is amplified when predictors are correlated. Orthogonal factorization solves the same least-squares problem without this loss of accuracy.

What does the condition number tell a statistician?

It measures how much small perturbations in the data can change the solution. A large condition number, typically from collinear predictors, warns that coefficient estimates are numerically and statistically unstable.

Numerical Linear Algebra for Statistics

Numerical linear algebra for statistics is the study of how the matrix computations underlying regression, multivariate analysis and covariance estimation are carried out accurately and efficiently in finite precision.

Definition

Numerical linear algebra for statistics is the application and analysis of finite-precision matrix algorithms to the linear-algebraic problems of statistics, principally least squares, covariance computation and the solution of linear systems arising in estimation.

Scope

This topic covers the solution of least-squares problems and normal equations, the conditioning of design matrices and its statistical consequences, the use of orthogonal methods for stability, and the efficient handling of large or structured covariance and design matrices. It is the statistical specialization of computational linear algebra; the matrix decompositions themselves are treated in a sibling topic.

Core questions

How are least-squares estimates computed accurately when predictors are nearly collinear?
Why are the normal equations numerically inferior to orthogonal approaches?
How does the conditioning of the design matrix affect estimated coefficients?
How are large and structured statistical matrices computed with efficiently?

Key concepts

Normal equations
Condition number
Collinearity
Orthogonalization
Backward stability

Key theories

Stable least squares: Solving least squares through orthogonal factorization avoids forming the normal equations, whose conditioning is the square of the original problem's, thereby preserving accuracy when predictors are correlated.
Conditioning and collinearity: Near-collinearity inflates the condition number of the design matrix, amplifying rounding error and the variance of estimated coefficients, which links a numerical property directly to statistical instability.

Clinical relevance

Accurate matrix computation determines whether regression coefficients, generalized-least-squares fits and covariance matrices are trustworthy; recognizing ill-conditioning explains otherwise puzzling instability in estimates and guides remedies such as centering, scaling or regularization.

History

The mid-twentieth-century development of numerically stable matrix algorithms by Wilkinson, Golub and others was steadily adopted by statisticians, who recognized that the normal-equations approach to regression was numerically fragile and adopted orthogonal alternatives.

Key figures

Gene Golub
Charles Van Loan
Kenneth Lange
James Wilkinson

Seminal works

golub2013
lange2010

Frequently asked questions

Why are the normal equations discouraged for least squares?: Forming the normal equations squares the condition number of the problem, so rounding error is amplified when predictors are correlated. Orthogonal factorization solves the same least-squares problem without this loss of accuracy.
What does the condition number tell a statistician?: It measures how much small perturbations in the data can change the solution. A large condition number, typically from collinear predictors, warns that coefficient estimates are numerically and statistically unstable.