Bayesian Inference Foundations
Bayesian inference treats unknown quantities as random variables and uses probability as the single calculus for representing and updating uncertainty in light of data.
Definition
Bayesian inference is the use of Bayes' theorem to convert a prior probability distribution over unknown parameters, combined with a likelihood for the observed data, into a posterior distribution that quantifies all remaining uncertainty about those unknowns.
Scope
This area covers the conceptual and mathematical core of the Bayesian approach: Bayes' theorem as an updating rule, the role of the likelihood, the interpretation of probability as degree of belief, exchangeability as the justification for statistical modeling, and the contrast between subjective and objective stances. It frames the rest of Bayesian statistics, which builds priors, computation, and models on top of these principles.
Sub-topics
Core questions
- How does Bayes' theorem combine prior beliefs with observed data to yield a posterior distribution?
- What role does the likelihood play, and why does Bayesian inference obey the likelihood principle?
- How does exchangeability justify representing observations as conditionally independent given parameters?
- What is the difference between subjective and objective interpretations of Bayesian probability?
Key concepts
- prior distribution
- likelihood function
- posterior distribution
- marginal likelihood (evidence)
- exchangeability
- coherence
- likelihood principle
Key theories
- Bayes' theorem as inference
- The posterior is proportional to the likelihood times the prior; this single identity governs how a Bayesian rationally updates uncertainty after observing data.
- De Finetti's representation theorem
- An infinite exchangeable sequence can be represented as conditionally i.i.d. given an unknown parameter with a mixing distribution, supplying a subjective-probability foundation for parametric models and priors.
- Coherence and the Dutch-book argument
- Degrees of belief that obey the probability axioms are 'coherent', avoiding sure-loss betting configurations; this decision-theoretic argument underpins the Bayesian use of probability for beliefs.
Clinical relevance
Bayesian foundations underpin applications across the sciences wherever uncertainty must be quantified and updated as evidence accumulates, from clinical trial monitoring and genetics to physics, machine learning, and decision analysis.
History
Bayes' essay (published posthumously in 1763) and Laplace's independent development gave the inverse-probability method its start. Eclipsed by frequentist methods in the early 20th century, the approach was revived through Jeffreys' objective priors, de Finetti's and Savage's subjective-probability foundations, and, from the 1990s, computational advances that made it broadly practical.
Debates
- Subjective versus objective priors
- Whether priors should encode genuine personal belief or be chosen by formal rules to minimize their influence remains a foundational dispute within Bayesian statistics.
Key figures
- Thomas Bayes
- Pierre-Simon Laplace
- Bruno de Finetti
- Harold Jeffreys
- Leonard J. Savage
Related topics
Seminal works
- gelman2013
- robert2007
- definetti1937
Frequently asked questions
- How is Bayesian inference different from frequentist inference?
- Bayesian inference assigns probability distributions to unknown parameters and reports a posterior distribution, whereas frequentist inference treats parameters as fixed and reasons about the long-run behavior of estimators and procedures over hypothetical repeated samples.
- Does Bayesian inference require a subjective prior?
- It requires a prior, but the prior can be subjective (encoding real belief) or chosen by objective rules to be weakly informative; with enough data the likelihood typically dominates and the choice matters less.