Gibbs Sampling

Gibbs sampling explores a posterior by updating each parameter in turn from its full conditional distribution given all the others.

Definition

Gibbs sampling is an MCMC method that cycles through the components of the parameter vector, drawing each from its conditional posterior distribution given the current values of all other components, producing a chain whose stationary distribution is the joint posterior.

Scope

This topic covers the full-conditional updates that define the Gibbs sampler, its status as a special case of Metropolis-Hastings with acceptance probability one, the use of data augmentation to create tractable conditionals, and blocking and collapsing strategies that improve mixing.

Core questions

• What are full conditional distributions and how are they used in Gibbs sampling?
• Why is Gibbs sampling a special case of Metropolis-Hastings?
• How does data augmentation create tractable conditionals?
• How do blocking and collapsing improve the sampler's efficiency?

Key concepts

• full conditional distribution
• data augmentation
• blocking
• collapsing
• latent variables
• componentwise updating

Key theories

Full-conditional updating

Sampling each parameter from its full conditional leaves the joint posterior invariant; when conditionals are conjugate the updates are closed-form and acceptance is automatic.

Data augmentation

Introducing latent variables can make otherwise intractable conditionals standard, turning hard problems such as mixtures and probit models into straightforward Gibbs updates.

Clinical relevance

Gibbs sampling made hierarchical and latent-variable models routine, and it underlies widely used software such as BUGS and JAGS for applied Bayesian modeling in biostatistics and the social sciences.

History

Geman and Geman introduced the Gibbs sampler in 1984 for image restoration, naming it after Gibbs distributions in statistical physics. Gelfand and Smith's 1990 paper showed its broad applicability to Bayesian inference, sparking widespread adoption.

Debates

Slow mixing under strong dependence

Componentwise Gibbs updates can mix poorly when parameters are highly correlated, motivating reparameterization, blocking, or gradient-based alternatives.

Key figures

• Stuart Geman
• Donald Geman
• Alan Gelfand
• Adrian Smith

Seminal works

• geman1984
• gelfand1990

Frequently asked questions

When is Gibbs sampling a good choice?

Gibbs sampling is well suited to models with conjugate or otherwise standard full conditionals, such as many hierarchical and latent-variable models, but it can mix slowly when parameters are strongly correlated.