Robust Variational Inference
Also known as: RVI, robust VI, outlier-robust variational Bayes, power-divergence variational inference
Robust variational inference (RVI) extends standard variational inference by replacing the Kullback-Leibler divergence with a divergence measure that is less sensitive to outliers and model misspecification — such as the beta-divergence or a Renyi-type divergence. This yields posterior approximations that remain well-behaved even when a fraction of the data departs from the assumed model.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use robust variational inference when your data set is likely to contain outliers, recording errors, or a proportion of observations that violate the core model assumptions, and when a full MCMC sampler is too slow for the data size or model complexity. It is particularly valuable in large-scale Bayesian deep learning and probabilistic modelling where model misspecification is expected. Avoid it when the data are clean and well-described by the assumed likelihood — the robustness tuning parameter then introduces unnecessary bias — or when exact posterior uncertainty is required and computation budget permits MCMC.
Strengths & limitations
- Inherits the scalability of standard variational inference while adding resistance to outliers and model misspecification.
- The robustness-efficiency trade-off is controlled by a single tuning parameter, allowing practitioners to dial in as much robustness as needed.
- Compatible with modern probabilistic programming tools and stochastic gradient optimisation pipelines.
- Produces calibrated posterior approximations for the bulk of the data even when a fraction of observations are anomalous.
- Applicable to complex hierarchical and deep probabilistic models where MCMC is computationally prohibitive.
- Choosing the robustness tuning parameter requires cross-validation or additional diagnostics; a poor choice either wastes statistical efficiency or fails to protect against outliers.
- The robust ELBO is not a true lower bound on the marginal likelihood under all divergences, complicating model comparison via ELBO values.
- Theory and software support are less mature than for standard KL-based VI, so bespoke implementation may be required.
- Robustness does not address all forms of model misspecification — structural misspecification (wrong functional form) requires model revision, not just a robust objective.
Frequently asked
How does robust VI differ from standard variational inference?
Standard VI minimises the KL divergence, which can be dominated by a small number of outlying observations. Robust VI replaces KL with a bounded divergence (e.g. beta-divergence) that limits the influence of any single observation, so the resulting posterior approximation reflects the majority of the data rather than its extremes.
How do I choose the robustness tuning parameter beta?
A common strategy is to select beta by cross-validated predictive performance on a held-out set, or to use a beta near zero (close to standard VI) and increase it only if diagnostic tools reveal outlier sensitivity. The choice is application-dependent: heavily contaminated data may need beta around 0.5, while clean data warrants beta close to zero.
Can I use robust VI with probabilistic programming languages like Stan or Pyro?
Stan's optimiser (variational Bayes) is KL-based and does not support custom divergences natively. Pyro and NumPyro allow custom ELBOs, so robust VI can be implemented by replacing the log-likelihood term. Libraries such as BayesFlow and custom PyTorch or JAX code are currently the most flexible options.
Is robust VI suitable for small samples?
With small samples the prior is already influential, which provides some inherent robustness. Robust VI adds a second layer of protection but at the cost of additional tuning. For very small n, robust Bayesian inference via MCMC with heavy-tailed likelihoods may be more interpretable.
Does robust VI guarantee better predictions than standard VI?
Only when the data are genuinely contaminated. On clean data, robust VI with beta > 0 introduces a small bias that can hurt predictive accuracy. Always compare robust and standard VI on a held-out set before committing to the robust objective.
Sources
- Futami, F., Sato, I. & Sugiyama, M. (2018). Variational inference based on robust divergences. Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 84:813-822. link ↗
- Ghosh, S. & Basu, A. (2016). Robust Bayes estimation using the density power divergence. Annals of the Institute of Statistical Mathematics, 68(2), 413-437. link ↗
How to cite this page
ScholarGate. (2026, June 3). Robust Variational Inference. ScholarGate. https://scholargate.app/en/bayesian/robust-variational-inference
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Approximate Bayesian ComputationSimulation↔ compare
- Bayesian RegressionBayesian↔ compare
- Markov Chain Monte CarloSimulation↔ compare
- Robust Bayesian InferenceBayesian↔ compare
- Robust Markov chain Monte CarloBayesian↔ compare
- Variational InferenceBayesian↔ compare