Expectation Propagation (EP)
Expectation Propagation for Approximate Bayesian Inference · Also known as: EP, expectation propagation, EP algorithm, assumed-density filtering generalisation
Expectation Propagation (EP) is a deterministic message-passing algorithm for approximate posterior inference in Bayesian models, introduced by Thomas P. Minka at UAI 2001. It iteratively refines a set of local approximate factors — each drawn from the exponential family — so that their product closely matches the true intractable posterior, achieving higher accuracy than mean-field variational inference on many probabilistic machine learning tasks.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use EP when the posterior is analytically intractable and you need a more accurate approximation than mean-field variational inference can provide — especially when the true posterior has significant marginal correlations that mean-field would incorrectly ignore. EP is particularly well-suited to models with likelihood terms that are non-conjugate to a Gaussian prior but still admit tractable moment integrals: probit/logit regression, Gaussian process classification, Student-t robust regression, and sparse Bayesian models. It is a deterministic method, so it is faster and more reproducible than MCMC, though it requires moment integrals to be computable in closed form or by quadrature. EP is not the right choice when the posterior is genuinely multimodal in ways that a unimodal exponential-family approximation cannot capture.
Strengths & limitations
- More accurate than mean-field variational inference for many models because EP matches moments rather than minimising the reverse KL that encourages under-dispersion.
- Deterministic and typically faster to converge than MCMC while providing uncertainty estimates that MCMC-level methods would agree with on well-behaved posteriors.
- Provides a consistent approximation to the marginal likelihood (model evidence) as a by-product of the normalisation constants Z_i, enabling Bayesian model comparison.
- Highly modular: each factor is updated independently, so new likelihood terms can be incorporated without redesigning the entire algorithm.
- Convergence is not guaranteed; EP can fail to converge or oscillate, particularly for models with strongly multi-modal or heavy-tailed posteriors.
- Requires that moment integrals of the tilted distribution be computable — analytically or by low-dimensional numerical quadrature — which restricts the class of applicable models.
- The exponential-family constraint forces q to be unimodal (e.g. Gaussian), so EP cannot represent genuinely multi-modal posteriors.
- Can be sensitive to initialisation and update ordering; convergence behaviour must be monitored carefully in practice.
Frequently asked
How does EP differ from mean-field variational inference?
Mean-field VI assumes all latent variables are mutually independent in the approximation and minimises the KL divergence KL(q || p), which encourages the approximation to under-cover the true posterior's tails and ignore correlations. EP instead minimises KL(p̃ || q) locally for each factor, which matches moments and tends to over-cover the posterior, producing more accurate marginal variances. In practice EP is more accurate than mean-field for many models but less straightforward to implement and not always convergent.
Is EP guaranteed to converge?
No. Unlike mean-field VI, which optimises a global evidence lower bound and therefore always converges to a local minimum, EP has no global convergence guarantee. It can oscillate or produce negative (invalid) variances in some models. Power EP (Minka 2004) introduces a damping parameter α that improves robustness. In practice, damped updates and careful monitoring of factor parameters are used to manage convergence.
What does EP give me besides the approximate posterior?
EP tracks the normalisation constants Z_i for each factor update. Their product approximates the marginal likelihood p(data) — the model evidence — which is the quantity used in Bayesian model selection and hyperparameter optimisation. This is a significant practical advantage over MCMC, which requires expensive thermodynamic integration to estimate the marginal likelihood.
When should I prefer MCMC over EP?
Choose MCMC when the posterior is genuinely multimodal or when you need exact (asymptotically) posterior samples rather than an approximation. EP is deterministic and faster but constrained to unimodal exponential-family approximations. For high-stakes inference where approximation quality is uncertain, MCMC provides a ground-truth reference against which EP results can be validated.
Sources
- Minka, T. P. (2001). Expectation propagation for approximate Bayesian inference. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI-01), pp. 362–369. Morgan Kaufmann. link ↗
- Minka, T. P. (2001/2013). Expectation propagation for approximate Bayesian inference. arXiv:1301.2294 [cs.AI]. link ↗
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. (Chapter 10: Approximate Inference; Section 10.7 covers Expectation Propagation.) ISBN: 978-0387310732
How to cite this page
ScholarGate. (2026, June 3). Expectation Propagation for Approximate Bayesian Inference. ScholarGate. https://scholargate.app/en/bayesian/expectation-propagation
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Laplace ApproximationBayesian↔ compare
- MCMCBayesian↔ compare
- Variational InferenceBayesian↔ compare