Classification and Discriminant Analysis

Classification and discriminant analysis comprises the multivariate methods that assign observations to predefined groups using measured features and a sample of labeled cases.

Definition

Discriminant analysis and classification are the construction of rules that assign a multivariate observation to one of several known groups so as to minimize an expected cost or error of misclassification.

Scope

This area covers supervised classification of multivariate observations. It includes Fisher's linear discriminant and its Gaussian-model interpretation, quadratic discriminant analysis for unequal group covariances, logistic discrimination as a direct model of class membership probabilities, and margin-based methods such as support vector machines. The focus is on the construction, geometry, and evaluation of decision boundaries.

Sub-topics

• Linear Discriminant Analysis
• Logistic Discrimination
• Quadratic Discriminant Analysis
• Support Vector Classification

Core questions

• How should an observation be assigned to one of several known groups from its measured features?
• What decision boundary minimizes the expected misclassification cost?
• When are linear boundaries adequate and when are quadratic or nonlinear boundaries needed?
• How is classifier performance estimated without optimistic bias?

Key theories

Bayes-optimal classification

Assigning each observation to the group with the highest posterior probability minimizes the expected misclassification error; parametric discriminant methods estimate these posteriors under distributional assumptions.

Fisher's linear discriminant

Fisher sought the linear combination of features that maximally separates group means relative to within-group spread, yielding a discriminant direction that, under equal Gaussian covariances, coincides with the Bayes rule.

Clinical relevance

Classification methods are used wherever cases must be sorted into known categories from multivariate measurements, including medical diagnosis, credit scoring, species identification, and remote-sensing land-cover mapping.

History

The field began with Fisher's 1936 linear discriminant applied to taxonomic measurements. Probabilistic and Gaussian formulations followed, logistic discrimination provided a direct model of class probabilities, and the late-twentieth-century development of margin-based and kernel methods extended classification to high-dimensional and nonlinear settings.

Debates

Generative versus discriminative classification

Generative methods such as discriminant analysis model the feature distribution within each class, while discriminative methods such as logistic regression and support vector machines model the boundary or class probability directly; their relative merits depend on sample size and how well distributional assumptions hold.

Key figures

• Ronald A. Fisher
• Vladimir Vapnik

Seminal works

• fisher1936
• hastie2009
• johnson2007

Frequently asked questions

How does classification differ from clustering?

Classification is supervised: the groups are known in advance and a labeled training sample is available. Clustering is unsupervised and discovers groupings without predefined labels.

Why estimate error on held-out data?

Error measured on the same data used to fit a classifier is optimistically biased, so out-of-sample estimates from cross-validation or a test set are needed to assess true predictive performance.