Gaussian Process
Gaussian Process Regression and Classification · Also known as: GP, Gaussian Process Regression, GPR, Kriging
A Gaussian Process (GP) is a non-parametric, fully probabilistic machine learning model that places a prior distribution directly over functions. Rather than predicting a single value, it returns a predictive mean and a calibrated uncertainty estimate at every test point, making it especially valuable for regression on small to medium datasets and for Bayesian optimization tasks.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+20 more
When to use it
Gaussian Processes excel on small to medium datasets (up to a few thousand rows) where uncertainty quantification matters — for example, scientific experiments, engineering design, or surrogate modelling for expensive simulations. They are the method of choice for Bayesian optimization. Prefer GP over simpler models when you need calibrated prediction intervals rather than just point estimates, and when the relationship between inputs and outputs is expected to be smooth. Avoid GP when n exceeds roughly 10,000 observations, as exact inference scales as O(n^3) making it computationally prohibitive without approximations. GP also struggles with very high-dimensional input spaces and with discontinuous or highly non-stationary functions unless carefully tailored kernels are used.
Strengths & limitations
- Returns calibrated predictive uncertainty intervals alongside point predictions.
- Extremely sample-efficient — performs well on small datasets where tree-based models overfit.
- Kernel choice flexibly encodes domain knowledge about smoothness, periodicity, or scale.
- Hyperparameters are learned from data via marginal likelihood, providing automatic regularisation.
- Native framework for Bayesian optimization and active learning experimental design.
- Exact inference is O(n^3) in time and O(n^2) in memory, making it impractical for large n without sparse or approximate GP methods.
- Performance degrades in very high-dimensional input spaces due to the curse of dimensionality in kernel distances.
- Choosing and combining kernels requires domain knowledge; a poorly chosen kernel leads to poor uncertainty calibration.
- Classification (GP classifiers) requires an additional Laplace or expectation-propagation approximation, adding complexity.
Frequently asked
What kernel should I start with?
The squared-exponential (RBF) kernel is the default starting point because it produces infinitely differentiable, smooth functions and has interpretable length-scale and output-variance hyperparameters. Switch to a Matern kernel if you expect the function to be less smooth, or add a periodic component for seasonal data.
My dataset has 50,000 rows. Can I still use a GP?
Exact GP inference would be prohibitive at that scale. Consider sparse GP methods (e.g., inducing points / FITC) or stochastic variational GP, both of which bring training cost down to roughly O(nm^2) where m is the number of inducing points (m << n). Libraries such as GPyTorch implement these efficiently.
How is a GP different from Bayesian linear regression?
Bayesian linear regression places a prior over a fixed finite set of coefficients; the model's flexibility is therefore limited by the chosen basis functions. A GP places a prior directly over functions in an infinite-dimensional space, so its effective complexity adapts to the data without requiring you to pre-specify basis expansions.
Are the uncertainty estimates truly calibrated?
Under the correct model and kernel, GP posterior uncertainties are well-calibrated by construction. In practice, miscalibration arises from kernel misspecification or ignoring noise structure. Always perform a posterior predictive check — plot observed vs. predicted with coverage of the credible intervals.
Can Gaussian Processes handle categorical or mixed-type inputs?
Standard GP kernels operate on continuous inputs. Categorical inputs require special kernels (e.g., Hamming distance kernel) or encoding. For predominantly categorical or very high-dimensional mixed data, tree-based models such as Random Forest or gradient boosting are usually a better starting point.
Sources
- Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press. ISBN: 978-0-262-18253-9
- Gaussian process. Wikipedia. link ↗
How to cite this page
ScholarGate. (2026, June 3). Gaussian Process Regression and Classification. ScholarGate. https://scholargate.app/en/machine-learning/gaussian-process
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Bayesian Gaussian ProcessMachine learning↔ compare
- Bayesian OptimizationOptimization↔ compare
- Random ForestMachine learning↔ compare