Gaussian Process Models

A Gaussian process places a prior directly on functions, so that regression and classification can be performed nonparametrically with calibrated uncertainty.

Definition

A Gaussian process is a distribution over functions such that the values at any finite set of inputs follow a multivariate normal distribution determined by a mean function and a covariance kernel; conditioning on observed data gives a posterior over functions used for prediction.

Scope

This topic covers the definition of a Gaussian process through its mean and covariance (kernel) functions, the closed-form posterior for regression, the role of kernel choice and hyperparameters in encoding smoothness, classification via latent Gaussian processes, and the computational cost of large datasets.

Core questions

• How does a covariance kernel define a prior over functions?
• How is the Gaussian process regression posterior computed in closed form?
• How do kernel hyperparameters control smoothness and length scale?
• What makes exact Gaussian process inference expensive for large datasets?

Key concepts

• covariance kernel
• mean function
• length scale
• Gaussian process regression
• latent Gaussian process
• marginal likelihood
• scalability

Key theories

Function-space prior

Specifying mean and covariance functions defines a coherent prior over functions; for Gaussian likelihoods the posterior mean and variance have closed forms given by the kernel matrix.

Neural-network limit

Neal showed that a single-layer neural network with infinitely many hidden units and suitable priors converges to a Gaussian process, linking Bayesian neural networks to Gaussian-process priors.

Clinical relevance

Gaussian processes provide flexible regression with uncertainty for spatial statistics, computer-model emulation, time-series interpolation, and Bayesian optimization across the sciences and engineering.

History

Gaussian-process regression has roots in kriging in geostatistics and in O'Hagan's work on curve fitting. Neal's 1996 connection to neural networks and the 2006 monograph by Rasmussen and Williams established Gaussian processes as a central machine-learning and nonparametric Bayesian tool.

Debates

Scaling to large data

Exact inference costs grow cubically with the number of observations, so much research concerns sparse and approximate methods that trade accuracy for scalability.

Key figures

• Carl Edward Rasmussen
• Christopher Williams
• Radford Neal
• Anthony O'Hagan

Seminal works

• rasmussen2006
• neal1996

Frequently asked questions

What does the kernel do in a Gaussian process?

The kernel sets the covariance between function values at different inputs, encoding assumptions such as smoothness and characteristic length scale; its choice and hyperparameters largely determine the shape and flexibility of the inferred function.