Gibbs Sampling
Gibbs Sampling Markov Chain Monte Carlo · Also known as: Gibbs sampler, coordinate-wise MCMC, systematic scan Gibbs, blocked Gibbs sampling
Gibbs sampling is a Markov chain Monte Carlo algorithm that approximates a high-dimensional posterior distribution by repeatedly drawing each parameter from its full conditional distribution given all other parameters and the data. Because each draw is exact from a conditional — not a proposal that may be rejected — the sampler is efficient when those conditionals are available in closed form.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+23 more
When to use it
Use Gibbs sampling when the joint posterior is intractable but all full conditional distributions can be derived in closed form — the typical situation with conjugate Bayesian models such as Bayesian linear regression with Normal-Inverse-Gamma priors, Gaussian mixture models, and Latent Dirichlet Allocation. It is also useful when parameters naturally group into blocks with tractable conditionals, allowing a blocked Gibbs strategy. Avoid Gibbs sampling when full conditionals cannot be computed analytically (use Metropolis-Hastings or HMC instead), when parameters are highly correlated in the posterior (causing slow mixing and requiring many more iterations), or when the parameter space is very high-dimensional with non-conjugate likelihoods.
Strengths & limitations
- Every proposed draw is accepted, so there is no wasted computation from rejections as in Metropolis-Hastings.
- Simple to implement for conjugate models: each step is a direct draw from a named distribution.
- Scales gracefully to models with many conditionally independent parameters (e.g., topic models, state-space models).
- Blocked Gibbs can group correlated parameters, substantially improving mixing.
- Conceptually transparent: the algorithm maps directly onto the model's conditional structure.
- Requires analytically tractable full conditional distributions; non-conjugate models demand auxiliary-variable augmentation or hybrid methods.
- Highly correlated parameters cause slow mixing: the chain takes many small steps and requires far more iterations to explore the posterior.
- Serial nature of the coordinate-wise update can be slow in very high dimensions compared to gradient-based samplers like HMC.
- Convergence can be hard to detect; poorly initialised or poorly mixed chains yield biased posterior summaries.
Frequently asked
Why does Gibbs sampling always accept? Is it not too good to be true?
Gibbs draws exactly from the correct conditional distribution, so the acceptance probability is always 1 by construction — there is no approximation in the draw itself. The caveat is that this only works when the full conditional is analytically available and can be sampled efficiently; when it cannot, you must fall back to a Metropolis step within the sampler.
How is Gibbs sampling related to Metropolis-Hastings?
Gibbs sampling is a special case of Metropolis-Hastings where each proposal is drawn exactly from the conditional, giving an acceptance ratio of exactly 1. In practice the two are often combined: a Metropolis-within-Gibbs scheme uses Metropolis steps for parameters whose conditionals are not standard distributions.
How many burn-in iterations do I need?
There is no universal rule. Run multiple chains from dispersed starting points and monitor R-hat (values below ~1.01 are reassuring) and the trace plots. Burn-in of a few hundred to a few thousand iterations is typical for well-conditioned models, but strongly correlated or poorly specified models may need tens of thousands.
When should I use HMC instead of Gibbs?
Prefer Hamiltonian Monte Carlo when your model has non-conjugate likelihoods, strong posterior correlations, or a continuous high-dimensional parameter space. HMC uses gradient information to make large, informed jumps, mixing far more efficiently in those settings. Gibbs remains preferable when conjugate conditionals are available and the model structure naturally decomposes into tractable blocks.
What software implements Gibbs sampling?
JAGS (Just Another Gibbs Sampler) and WinBUGS/OpenBUGS are dedicated Gibbs-based samplers widely used in applied Bayesian statistics. Stan uses HMC/NUTS but also allows custom sampler logic. In Python, PyMC and NumPyro offer Gibbs steps alongside other samplers; in R, the rjags and R2jags packages interface with JAGS.
Sources
- Geman, S. & Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(6), 721-741. DOI: 10.1109/TPAMI.1984.4767596 ↗
- Gelfand, A. E. & Smith, A. F. M. (1990). Sampling-based approaches to calculating marginal densities. Journal of the American Statistical Association, 85(410), 398-409. DOI: 10.1080/01621459.1990.10476213 ↗
How to cite this page
ScholarGate. (2026, June 3). Gibbs Sampling Markov Chain Monte Carlo. ScholarGate. https://scholargate.app/en/bayesian/gibbs-sampling
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Bayesian RegressionBayesian↔ compare
- Hamiltonian Monte CarloBayesian↔ compare
- Hierarchical Bayesian InferenceBayesian↔ compare
- MCMCBayesian↔ compare
- Variational InferenceBayesian↔ compare