How does PCA differ from factor analysis?

PCA forms components as exact linear combinations that maximize variance, with no explicit error model, whereas factor analysis posits latent common factors plus variable-specific noise to explain shared covariance.

Should variables be standardized before PCA?

When variables are on different scales it is common to standardize, which is equivalent to performing PCA on the correlation matrix, so that no single high-variance variable dominates the components.

Principal Component Analysis

Principal component analysis (PCA) finds an orthogonal set of derived variables, the principal components, that successively capture the maximum possible variance in a multivariate dataset.

Atrast tematu ar PaperMindDrīzumāFind papers & topics

Tools & resources

Lejupielādēt slaidus

Learn & explore

VideoDrīzumā

Definition

Principal component analysis is an orthogonal linear transformation that re-expresses correlated variables as uncorrelated components ordered so that the first component has the largest variance and each successive component the largest variance orthogonal to the preceding ones.

Scope

This topic covers the definition of principal components as eigenvectors of the covariance or correlation matrix, their equivalence to the best low-rank least-squares approximation via the singular value decomposition, the interpretation of loadings and scores, choice of the number of components, and the distinction between covariance-based and correlation-based analyses.

Core questions

Which linear combinations of variables explain the most variance?
How many components are required to summarize the data adequately?
When should the analysis be based on the correlation rather than the covariance matrix?
How are component loadings and scores interpreted?

Key theories

Eigen-decomposition of the covariance matrix: The principal components are the eigenvectors of the covariance matrix, and the variance explained by each component equals the corresponding eigenvalue, giving an exact orthogonal decomposition of total variance.
Best low-rank approximation: Projecting data onto the leading principal axes minimizes the sum of squared reconstruction errors among all subspaces of that dimension, the property Pearson originally formulated as lines and planes of closest fit.

Clinical relevance

PCA is widely used for visualization, denoising, compression, multicollinearity diagnosis, and as a preprocessing step that produces uncorrelated features for regression and classification.

History

Pearson introduced the geometric idea of best-fitting lines and planes in 1901; Hotelling independently developed and named principal components as a statistical technique in 1933. The method was later unified with the singular value decomposition, which provides its standard numerical implementation.

Debates

Choosing the number of components: Rules such as retaining components with eigenvalues above one, inspecting the scree plot, or fixing a cumulative-variance threshold can disagree, and no single criterion is universally accepted.

Key figures

Karl Pearson
Harold Hotelling

Seminal works

pearson1901
hotelling1933
jolliffe2002

Frequently asked questions

How does PCA differ from factor analysis?: PCA forms components as exact linear combinations that maximize variance, with no explicit error model, whereas factor analysis posits latent common factors plus variable-specific noise to explain shared covariance.
Should variables be standardized before PCA?: When variables are on different scales it is common to standardize, which is equivalent to performing PCA on the correlation matrix, so that no single high-variance variable dominates the components.