Bayesian Inference
Also known as: Bayes inference, Bayesian statistics, Bayesian updating, posterior inference, Bayesian data analysis
Bayesian inference is a statistical paradigm in which probability represents degrees of belief rather than long-run frequencies. It encodes prior knowledge about parameters in a prior distribution, combines that prior with the likelihood of observed data via Bayes' theorem, and produces a posterior distribution that quantifies updated uncertainty. The foundational theorem was published posthumously by Thomas Bayes in 1763 and subsequently systematized by Pierre-Simon Laplace in his 1812 Théorie analytique des probabilités.
Key highlights
- Produces a full posterior distribution over parameters, enabling direct probability statements and complete uncertainty quantification rather than binary accept/reject decisions.
- Incorporates prior knowledge formally, which is especially valuable in small samples, rare events, or settings where earlier experiments have been conducted.
- Handles hierarchical and multilevel models naturally, shrinking estimates toward group-level means in a principled way.
- Supports sequential updating: the posterior from one study becomes the prior for the next, making cumulative evidence synthesis straightforward.
- Prediction is based on the posterior predictive distribution, which averages over parameter uncertainty rather than conditioning on a single point estimate.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Bayesian inference is appropriate whenever prior information exists and should be incorporated transparently, when uncertainty must be fully propagated rather than summarized as a single p-value, when the sample is small and frequentist asymptotics are unreliable, or when hierarchical or multilevel structure is present in the data. It is also the natural framework for sequential updating — adding new data refines an existing posterior without restarting the analysis. Key modeling decisions include the choice of prior and the assumed likelihood family; both should be checked with prior and posterior predictive simulations. No single assumption of normality or equal variance is imposed — the distributional model is specified explicitly by the analyst.
Strengths & limitations
- Produces a full posterior distribution over parameters, enabling direct probability statements and complete uncertainty quantification rather than binary accept/reject decisions.
- Incorporates prior knowledge formally, which is especially valuable in small samples, rare events, or settings where earlier experiments have been conducted.
- Handles hierarchical and multilevel models naturally, shrinking estimates toward group-level means in a principled way.
- Supports sequential updating: the posterior from one study becomes the prior for the next, making cumulative evidence synthesis straightforward.
- Prediction is based on the posterior predictive distribution, which averages over parameter uncertainty rather than conditioning on a single point estimate.
- Results depend on the prior distribution; poorly chosen or unjustified priors can dominate inference, particularly with small samples.
- Computing the posterior is analytically intractable for most realistic models and requires MCMC, variational inference, or other approximation methods, which can be computationally expensive.
- Diagnosing MCMC convergence (R-hat, effective sample size, trace plots) and validating the model requires expertise and adds workflow overhead.
- Communicating probabilistic results and credible intervals to non-statistical audiences can be more challenging than reporting a p-value.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How does a Bayesian credible interval differ from a frequentist confidence interval?
A 95% Bayesian credible interval contains the parameter with 95% posterior probability — a direct probability statement about the parameter itself. A 95% frequentist confidence interval means that 95% of intervals constructed by the same procedure across repeated samples would contain the true parameter; the parameter is fixed and the interval is random. The two intervals can be numerically similar but their interpretations are fundamentally different.
Does Bayesian inference require a large sample to work?
No — this is one of its advantages. Frequentist asymptotics (e.g., the Central Limit Theorem guaranteeing normality of estimators) require large samples. Bayesian inference is exact for any sample size given the specified model; the posterior simply reflects more uncertainty when n is small. Informative priors become especially valuable in small-sample settings, as they prevent the posterior from wandering into implausible regions.
What is the difference between an informative and a non-informative prior?
An informative prior concentrates probability on a range of parameter values consistent with previous knowledge or domain expertise. A non-informative (or weakly informative) prior attempts to let the data dominate by spreading probability broadly. Fully flat priors are often improper (do not integrate to one) and can produce improper posteriors; weakly informative priors such as half-Normal or regularizing Normal distributions are preferred in practice (Gelman et al., 2013).
When should I use MCMC versus an analytical solution?
Analytical posterior solutions exist only for conjugate prior–likelihood pairs — for example, a Beta prior with a Binomial likelihood yields a Beta posterior. For most applied models, including regression, hierarchical models, and mixture models, the posterior is intractable and must be approximated. MCMC (Hamiltonian Monte Carlo via Stan, or the Metropolis–Hastings algorithm) provides asymptotically exact draws from the posterior; variational inference is faster but approximate. Use MCMC unless computation time is prohibitive.
Sources
- 1.Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53, 370–418.
- 2.Laplace, P.-S. (1812). Théorie analytique des probabilités. Courcier, Paris.
- 3.Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). Chapman & Hall/CRC.ISBN 978-1439840955
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Bayesian Inference. ScholarGate. https://scholargate.app/statistics/bayesian-inference