Kernel PCA
Kernel Principal Component Analysis · Also known as: KPCA, kernel PCA, nonlinear PCA via kernel trick, kernel eigenvalue decomposition
Kernel Principal Component Analysis (Kernel PCA) is a nonlinear dimensionality-reduction method introduced by Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller in 1997–1998. It extends classical linear PCA to curved, non-linear data manifolds by implicitly mapping input data into a high-dimensional feature space via a kernel function, then performing standard PCA in that space — all without ever computing the mapping explicitly.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Kernel PCA is appropriate when the data's variance structure is non-linear — for example when classes are arranged in concentric rings, on curved manifolds, or in configurations that standard PCA cannot separate. It is a natural choice as a preprocessing or feature-extraction step before a linear classifier when the raw feature space is not linearly separable. Key requirements and assumptions: (1) a kernel function that reflects the true similarity structure of the domain must be selected and tuned; (2) the method produces a read-only embedding — unlike PCA there is no closed-form inverse transform, making reconstruction non-trivial; (3) computational cost is O(n^3) for the eigendecomposition and O(n) per test-point projection, so it becomes expensive for very large datasets (n above roughly 10 000) unless approximations such as Nyström sampling are used; (4) the number of retained components d must be chosen, typically by examining the eigenvalue spectrum.
Strengths & limitations
- Captures nonlinear variance structure that linear PCA cannot reveal, enabling separation of geometrically complex data distributions.
- Exact kernelisation: the kernel trick means the method operates in arbitrarily high-dimensional feature spaces at the cost of n × n kernel evaluations, with no explicit feature-map computation.
- Kernel choice is flexible — RBF, polynomial, sigmoid, and custom domain kernels are all applicable, allowing the method to be tailored to specific data geometries.
- Out-of-sample extension is straightforward: new points are projected using the same kernel evaluations against training points without refitting.
- Serves as a theoretically principled bridge between linear methods (PCA, LDA) and nonlinear manifold-learning methods (Isomap, LLE, t-SNE).
- Computational and memory cost scales as O(n^2) in storage and O(n^3) in eigendecomposition, making exact Kernel PCA impractical for very large datasets without approximation.
- There is no closed-form pre-image (inverse) map, so reconstructing data in input space from kernel principal components is an ill-posed problem requiring iterative approximation.
- Kernel and hyperparameter selection (e.g., the bandwidth σ of an RBF kernel) is not automatic and can strongly affect the embedding quality; cross-validation is required.
- The method is unsupervised and has no direct mechanism to align components with class structure — supervised variants or follow-up classifiers are needed for discriminative tasks.
- Eigenvalues and components are not interpretable in input-space units, limiting direct scientific reporting of 'loadings' in the way classical PCA allows.
Frequently asked
Which kernel should I choose?
There is no universally best kernel. The Gaussian (RBF) kernel with bandwidth σ is the most common default and works well when the data's local geometry is approximately isotropic. Polynomial kernels suit data with known polynomial interactions. The best practical strategy is to try two or three kernels with cross-validated bandwidth/degree selection and compare the structure of the resulting embeddings visually and by downstream task performance.
How is Kernel PCA different from t-SNE or UMAP?
Kernel PCA produces a linear projection in the kernel feature space and supports exact out-of-sample extension via the kernel projection formula, making it reproducible and invertible in principle. t-SNE and UMAP optimise neighbourhood-preservation objectives and excel at visualisation but do not provide a fixed mapping for new points (without extensions) and are not designed for feature extraction preceding a classifier. Kernel PCA is more theoretically principled but typically less visually dramatic for cluster separation than t-SNE.
Can Kernel PCA reconstruct the original data?
Not directly. The projection into feature space is implicit and there is no closed-form pre-image in input space. Approximate pre-image methods exist (e.g., fixed-point iteration minimising residuals in feature space) but they are iterative and can be unstable. If reconstruction is a primary requirement, autoencoders or VAEs are better choices.
How do I handle large datasets?
Exact Kernel PCA requires storing and decomposing an n × n matrix, which is infeasible for n above roughly 10 000 on standard hardware. The Nyström approximation selects a landmark subset of m points (m ≪ n) and approximates the full kernel matrix from the m × n submatrix, reducing cost to O(m·n) in memory and O(m^2·n) in compute. Random Fourier features (Rahimi & Recht 2007) offer an alternative explicit low-dimensional approximation of shift-invariant kernels.
Sources
- Schölkopf, B., Smola, A. J., & Müller, K.-R. (1998). Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10(5), 1299–1319. DOI: 10.1162/089976698300017467 ↗
- Schölkopf, B., Smola, A. J., & Müller, K.-R. (1997). Kernel principal component analysis. In Artificial Neural Networks — ICANN'97, Lecture Notes in Computer Science, Vol. 1327, pp. 583–588. Springer. DOI: 10.1007/BFb0020217 ↗
- Schölkopf, B., & Smola, A. J. (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press. ISBN: 978-0-262-19475-4
How to cite this page
ScholarGate. (2026, June 3). Kernel Principal Component Analysis. ScholarGate. https://scholargate.app/en/machine-learning/kernel-pca
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- AutoencoderDeep learning↔ compare
- IsomapMachine learning↔ compare
- Locally Linear EmbeddingMachine learning↔ compare
- Support Vector MachineMachine learning↔ compare
- t-SNEMachine learning↔ compare