Variational Inference
Variational Bayesian Inference · Also known as: VI, variational Bayes, VB, mean-field variational inference, ELBO optimisation, variational approximation
Variational inference (VI) is a family of techniques that turn Bayesian posterior computation into an optimisation problem. Instead of drawing samples from the exact posterior — as Markov chain Monte Carlo does — VI posits a simpler, tractable family of distributions and finds the member of that family closest to the true posterior by maximising the evidence lower bound (ELBO). Introduced in its modern graphical-model form by Jordan, Ghahramani, Jaakkola and Saul (1999) and given a comprehensive statistical treatment by Blei, Kucukelbir and McAuliffe (2017), VI is now the standard scalable inference engine in probabilistic machine learning.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+15 more
When to use it
Variational inference is the right choice when (1) the posterior is intractable and MCMC would be too slow for the dataset size or model complexity; (2) the model involves continuous latent variables that admit a tractable ELBO under a chosen variational family; and (3) an approximate but fast answer is acceptable. It is routinely applied to topic models, deep generative models (variational autoencoders), Gaussian process approximations, and any hierarchical Bayesian model fit to large data. VI is not suited when the posterior has sharp multimodality or heavy tails that a unimodal or mean-field family cannot capture, or when exact posterior summaries are required.
Strengths & limitations
- Scales to large datasets through mini-batch stochastic optimisation, unlike MCMC.
- Provides a concrete scalar diagnostic — the ELBO — whose increase confirms the approximation is improving.
- Can be combined with automatic differentiation (automatic differentiation VI, ADVI) to handle a broad class of models without deriving update equations by hand.
- Produces a full approximate posterior distribution, not just a point estimate, preserving uncertainty quantification.
- The mean-field assumption ignores posterior correlations, which can cause variances to be underestimated and the approximation to be overconfident.
- KL(q || p) is asymmetric: it penalises q for placing mass where p is small, so VI tends to concentrate on one mode and miss others in multimodal posteriors.
- The ELBO is a lower bound, not the true log marginal likelihood, so it cannot directly be used for model comparison without correction.
- Convergence to a local maximum of the ELBO is not guaranteed; different initialisations may yield different solutions.
Frequently asked
How does variational inference differ from MCMC?
MCMC draws correlated samples that, given enough time, exactly represent the posterior. VI approximates the posterior with a simpler distribution found by optimisation. VI is faster and scales to larger data, but returns an approximation whose accuracy depends on how well the chosen variational family can represent the true posterior. MCMC is asymptotically exact but much slower for large models.
What is the ELBO and why does maximising it matter?
ELBO stands for Evidence Lower BOund. It equals log p(x) minus KL(q || p), so maximising the ELBO over q minimises the KL divergence between the approximate posterior q and the true posterior p. Because the log marginal likelihood log p(x) is constant in q, the ELBO is the only term that can be increased, making its maximisation the precise mathematical statement of finding the best approximation.
What is the mean-field assumption and when does it fail?
Mean-field assumes the approximate posterior factors into independent components, one per latent variable or group. This makes the ELBO tractable but ignores all correlations in the posterior. It fails when latent variables are genuinely correlated — for example, the slope and intercept in a regression posterior — leading to overconfident (too narrow) marginal approximations.
Can I use variational inference for model comparison?
The ELBO is a lower bound on the log marginal likelihood, so a higher ELBO for one model relative to another is evidence in its favour, but the gap between the bound and the true value can differ across models and variational families. Caution is required: if the bound is loose for one model, ELBO comparison can be misleading. Methods such as importance-weighted bounds (IWAE) can tighten the estimate.
Sources
- Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999). An introduction to variational methods for graphical models. Machine Learning, 37(2), 183–233. DOI: 10.1023/A:1007665907178 ↗
- Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017). Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518), 859–877. DOI: 10.1080/01621459.2017.1285773 ↗
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. (Chapter 10: Approximate Inference.) ISBN: 978-0387310732
How to cite this page
ScholarGate. (2026, June 3). Variational Bayesian Inference. ScholarGate. https://scholargate.app/en/bayesian/variational-inference
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Bayesian RegressionBayesian↔ compare
- Expectation PropagationBayesian↔ compare
- Latent Dirichlet AllocationMachine learning↔ compare
- MCMCBayesian↔ compare