Gibbs Sampling (Statistical Computing)
Gibbs sampling is a Markov chain Monte Carlo method that samples a multivariate distribution by cycling through its variables and drawing each in turn from its full conditional distribution given the current values of the others.
Definition
Gibbs sampling is a Markov chain Monte Carlo algorithm that generates draws from a joint distribution by iteratively sampling each component, or block of components, from its conditional distribution given all the others.
Scope
This topic treats Gibbs sampling as a computational algorithm: the construction of the sampler from full conditional distributions, its interpretation as a Metropolis-Hastings step with acceptance probability one, blocking and collapsing strategies that improve mixing, data augmentation, and the algorithm's convergence and autocorrelation behaviour. The applied Bayesian-inference perspective is covered separately under bayesian computation.
Core questions
- How do repeated draws from full conditional distributions converge to the joint target?
- Why is the Gibbs sampler a Metropolis-Hastings algorithm with acceptance probability one?
- How do blocking and collapsing improve the mixing of the sampler?
- How does data augmentation introduce latent variables to make conditionals tractable?
Key concepts
- Full conditional distribution
- Data augmentation
- Blocking and collapsing
- Compatibility of conditionals
- Mixing
Key theories
- Full-conditional sampling
- Sampling each variable in turn from its conditional distribution given the others defines a Markov chain whose stationary distribution is the joint target, provided the conditionals are compatible and the chain is irreducible.
- Data augmentation and blocking
- Introducing auxiliary latent variables can make full conditionals standard and easy to sample, while updating correlated variables in blocks reduces the autocorrelation that slow component-wise updates would otherwise produce.
Clinical relevance
Gibbs sampling is a workhorse algorithm of statistical computing because many models have simple, standard full conditionals; it underlies general-purpose samplers and is applied to mixed models, latent-variable models, image restoration and genetic linkage analysis.
History
Geman and Geman introduced the Gibbs sampler in 1984 for image restoration, naming it after the Gibbs distributions of statistical physics; Gelfand and Smith's 1990 paper showed its broad applicability, igniting widespread adoption across computational statistics.
Key figures
- Stuart Geman
- Donald Geman
- Alan Gelfand
- Adrian Smith
Related topics
Seminal works
- geman1984
- gelfand1990
Frequently asked questions
- When is Gibbs sampling especially convenient?
- When the full conditional distributions are standard distributions that can be sampled directly. Then no tuning of proposals or acceptance steps is needed because every proposal is accepted.
- Why can a Gibbs sampler mix slowly?
- When variables are highly correlated, updating them one at a time moves the chain in small steps along narrow ridges, producing high autocorrelation. Blocking correlated variables or reparameterizing the model can improve mixing.