Quadratic Discriminant Analysis

Quadratic discriminant analysis classifies observations under multivariate Gaussian groups that are allowed to have different covariance matrices, producing curved decision boundaries.

Definition

Quadratic discriminant analysis is a classification method that models each group as multivariate normal with its own covariance matrix and assigns observations by comparing quadratic discriminant scores derived from these densities.

Scope

This topic covers the Gaussian classification model with group-specific covariance matrices, the resulting quadratic discriminant functions, the trade-off in parameters relative to linear discriminant analysis, sensitivity to small samples, and regularized approaches that interpolate between the linear and quadratic rules.

Core questions

• How does relaxing the equal-covariance assumption change the decision boundary?
• When does the added flexibility of separate covariances improve classification?
• Why is quadratic discriminant analysis more prone to overfitting in small samples?
• How can regularization stabilize the covariance estimates?

Key theories

Unequal-covariance Gaussian model

When each group is multivariate normal with its own covariance matrix, the log-ratio of densities is quadratic in the features, so the Bayes-optimal boundary between groups is a quadratic surface.

Bias-variance trade-off with linear discriminant analysis

Quadratic discriminant analysis estimates a separate covariance per group, reducing bias when covariances truly differ but increasing variance, so it can be outperformed by the linear rule when samples are small.

Clinical relevance

Quadratic discriminant analysis is applied when groups plausibly differ in their dispersion as well as their means, offering more flexible boundaries than the linear rule in classification problems across science and engineering.

History

Quadratic discrimination arose as the natural extension of Fisher's and the Gaussian linear discriminant once the assumption of a common covariance matrix is dropped, and was later complemented by regularized discriminant analysis to handle high-dimensional and small-sample data.

Debates

Linear versus quadratic boundaries

Allowing group-specific covariances can capture genuinely curved boundaries but multiplies the number of estimated parameters, so the choice between linear and quadratic discriminant analysis is a bias-variance decision sensitive to sample size.

Key figures

• T. W. Anderson

Seminal works

• anderson2003
• hastie2009
• johnson2007

Frequently asked questions

When should I use QDA instead of LDA?

Use quadratic discriminant analysis when the groups appear to have substantially different covariance structures and the sample is large enough to estimate a separate covariance matrix per group reliably.

What is regularized discriminant analysis?

It is a compromise that shrinks each group covariance toward a pooled estimate, tuning a parameter that smoothly interpolates between quadratic and linear discriminant analysis.