When is linear discriminant analysis preferred over logistic regression?

When the Gaussian equal-covariance assumption holds reasonably well, especially in small samples or with well-separated classes, LDA can be more efficient; logistic regression is more robust when those assumptions are doubtful.

Can LDA reduce dimensionality?

Yes. With several groups it produces discriminant coordinates that span a lower-dimensional subspace maximizing group separation, which can be used for visualization.

Linear Discriminant Analysis

Linear discriminant analysis separates predefined groups using a linear combination of features, optimal when the groups are Gaussian with a common covariance matrix.

Definition

Linear discriminant analysis is a classification method that assigns an observation to the group whose mean is closest in Mahalanobis distance under a common covariance matrix, producing linear boundaries between groups.

Scope

This topic covers Fisher's criterion of maximizing between-group relative to within-group variance, the equivalent Gaussian model with equal covariances that yields linear decision boundaries, the role of the pooled covariance matrix, multi-group discrimination via discriminant coordinates, and the connection to the Mahalanobis distance.

Core questions

What linear combination of features best separates the groups?
Under what assumptions is the linear rule optimal?
How does the pooled within-group covariance enter the discriminant function?
How are several groups handled simultaneously?

Key theories

Maximization of separation: Fisher's discriminant chooses the projection direction that maximizes the ratio of between-group to within-group variance, giving the most separating linear combination of the features.
Equal-covariance Gaussian model: When groups are multivariate normal with a shared covariance matrix, the log-ratio of class densities is linear in the features, so the Bayes classifier reduces to the linear discriminant based on Mahalanobis distance to group means.

Clinical relevance

Linear discriminant analysis remains a simple and interpretable baseline classifier and a dimension-reduction tool that projects data onto the directions that best separate known groups, used in diagnostics, face recognition, and chemometrics.

History

Fisher introduced the linear discriminant in 1936 using measurements on iris flowers, framing it as a separation problem. Its equivalence to the Bayes rule for equal-covariance Gaussian populations was established subsequently, linking the geometric and probabilistic views.

Debates

Robustness of the equal-covariance assumption: Linear discriminant analysis assumes a common covariance across groups; when this fails, quadratic discriminant analysis or regularized variants may perform better, though the linear rule is often more stable in small samples.

Key figures

Ronald A. Fisher
P. C. Mahalanobis

Seminal works

fisher1936
anderson2003
hastie2009

Frequently asked questions

When is linear discriminant analysis preferred over logistic regression?: When the Gaussian equal-covariance assumption holds reasonably well, especially in small samples or with well-separated classes, LDA can be more efficient; logistic regression is more robust when those assumptions are doubtful.
Can LDA reduce dimensionality?: Yes. With several groups it produces discriminant coordinates that span a lower-dimensional subspace maximizing group separation, which can be used for visualization.