Automatic Differentiation Variational Inference (ADVI)
Also known as: ADVI, black-box variational inference, automatic variational inference, gradient-based variational inference
Automatic Differentiation Variational Inference (ADVI) is a black-box algorithm for approximate Bayesian posterior inference, introduced by Kucukelbir, Tran, Ranganath, Gelman, and Blei (2017, JMLR). Given any probabilistic model whose log-joint density is differentiable, ADVI automatically transforms constrained latent variables to unconstrained real space, fits a Gaussian variational family by maximising the evidence lower bound (ELBO) with stochastic gradient ascent, and returns an approximate posterior without model-specific derivations. It is the default variational inference engine in Stan and is available in PyMC and NumPyro.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
ADVI is appropriate when the posterior cannot be sampled with MCMC in reasonable time — typically because the model has many parameters, the data are large, or fast approximate inference is needed for iterative model development. It assumes all latent variables are continuous and that the log-joint density is differentiable; discrete latent variables require marginalisation or a separate approach. It is well-suited to hierarchical models, latent Gaussian models, and any model already implemented in Stan or PyMC. When exact or highly accurate posterior uncertainty is required, full MCMC (e.g. NUTS) should be preferred, as the mean-field Gaussian family can underestimate posterior variance and miss multimodal or strongly correlated posteriors.
Strengths & limitations
- Fully automatic: requires no model-specific derivations beyond specifying the log-joint density.
- Scales to large datasets through stochastic gradient estimation with minibatches.
- Orders of magnitude faster than MCMC for complex hierarchical models, enabling rapid iteration during model development.
- Built directly into Stan's variational inference engine and available in PyMC and NumPyro, making it accessible without custom code.
- The transformation step handles any combination of constrained and unconstrained parameters uniformly.
- The mean-field Gaussian variational family cannot capture posterior correlations between parameters; this leads to underestimated posterior variance in models with strong parameter dependencies.
- Convergence is to a local optimum of the ELBO, not the global one; results can depend on initialisation.
- Does not apply to models with discrete latent variables unless those variables are analytically marginalised.
- Provides no diagnostic analogous to MCMC's R-hat to certify that the approximation is close to the true posterior; the ELBO is a lower bound but its absolute value is not directly interpretable.
- The Gaussian approximation in transformed space may be a poor fit when the true posterior is heavy-tailed, multimodal, or highly skewed.
Frequently asked
How does ADVI differ from standard (coordinate-ascent) variational inference?
Coordinate-ascent variational inference (CAVI) requires manually deriving closed-form update equations for each parameter conditional on the others, which is possible only for specific model–prior combinations (conjugate exponential family models). ADVI replaces all of this with a single stochastic gradient step computed by automatic differentiation, making it applicable to any differentiable probabilistic model without model-specific derivations.
Why does ADVI transform parameters to unconstrained space?
The Gaussian variational family is defined on all of ℝ, but many model parameters are constrained (e.g. a variance must be positive). By applying a bijection such as the log transform, ADVI maps each constrained parameter to the real line, where a Gaussian approximation is valid. The Jacobian of the transformation corrects the density so that the approximation remains probabilistically consistent.
Can I trust ADVI's posterior uncertainty estimates?
The mean-field Gaussian family approximates each parameter independently, so posterior correlations between parameters are ignored. This typically causes ADVI to underestimate posterior variance, sometimes substantially. For critical inferences, results should be validated against MCMC on a smaller dataset or a simplified model version. The approximate means are often more reliable than the approximate variances.
When should I prefer MCMC (NUTS) over ADVI?
Prefer NUTS when exact posterior uncertainty is important, when the model has strong parameter correlations, when the posterior may be multimodal, or when the dataset is small enough that MCMC finishes quickly. ADVI is the practical choice when iteration speed matters (e.g. exploring many models) or when MCMC mixing times are too long for a large or complex model.
Sources
- Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A. & Blei, D. M. (2017). Automatic differentiation variational inference. Journal of Machine Learning Research, 18(14), 1–45. link ↗
- Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A. & Blei, D. M. (2016). Automatic differentiation variational inference. arXiv:1603.00788. link ↗
- Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A. & Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). CRC Press. ISBN: 978-1439840955
How to cite this page
ScholarGate. (2026, June 3). Automatic Differentiation Variational Inference (ADVI). ScholarGate. https://scholargate.app/en/bayesian/automatic-differentiation-variational-inference
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Bayesian RegressionBayesian↔ compare
- Expectation PropagationBayesian↔ compare
- MCMCBayesian↔ compare