Bayesian Model Comparison and Selection
Bayesian model comparison weighs competing models by how well they predict and how much posterior support the data give them, using marginal likelihoods, predictive criteria, and model averaging.
Definition
Bayesian model comparison is the use of probability to evaluate and choose among competing models, by comparing their marginal likelihoods or posterior probabilities, by estimating their expected predictive accuracy, or by averaging over them in proportion to their support.
Scope
This area covers Bayes factors and the marginal likelihood, predictive information criteria such as WAIC and leave-one-out cross-validation, Bayesian model averaging that accounts for model uncertainty, and posterior predictive checking for assessing absolute model fit.
Sub-topics
Core questions
- How do Bayes factors and posterior model probabilities compare models?
- How is expected predictive accuracy estimated using WAIC and cross-validation?
- How does Bayesian model averaging handle uncertainty about which model is correct?
- How do posterior predictive checks assess whether a single model fits the data?
Key concepts
- Bayes factor
- marginal likelihood
- WAIC
- leave-one-out cross-validation
- Bayesian model averaging
- posterior predictive check
- Occam's razor
- predictive accuracy
Key theories
- Bayes factors
- The ratio of marginal likelihoods quantifies the evidence the data provide for one model over another and is the formal Bayesian basis for hypothesis and model comparison.
- Predictive model evaluation
- Information criteria such as WAIC and efficient leave-one-out cross-validation estimate out-of-sample predictive accuracy directly from posterior draws, providing a prediction-focused alternative to Bayes factors.
Clinical relevance
Model comparison guides which scientific or predictive model to trust in fields from genetics to cosmology, and posterior predictive checks provide a principled way to detect model misfit before conclusions are drawn.
History
Jeffreys introduced Bayes factors for hypothesis testing in the 1930s; Kass and Raftery's 1995 review made them widely accessible. Concern about the marginal likelihood's sensitivity to priors and computation spurred predictive criteria such as DIC, WAIC, and efficient leave-one-out cross-validation.
Debates
- Bayes factors versus predictive criteria
- Bayes factors depend sensitively on the prior and can be hard to compute, while predictive criteria target out-of-sample accuracy; which to prefer depends on whether the goal is evidence for a hypothesis or predictive performance.
Key figures
- Harold Jeffreys
- Robert Kass
- Adrian Raftery
- Sumio Watanabe
- Aki Vehtari
Related topics
Seminal works
- kass1995
- vehtari2017
- gelman2013
Frequently asked questions
- Should I use Bayes factors or an information criterion?
- Use Bayes factors when you want a measure of evidence for one hypothesis over another and can specify priors carefully; use predictive criteria such as WAIC or leave-one-out cross-validation when the goal is to compare expected out-of-sample predictive performance.