Bayesian Model Comparison and Selection

Bayesian model comparison weighs competing models by how well they predict and how much posterior support the data give them, using marginal likelihoods, predictive criteria, and model averaging.

Definition

Bayesian model comparison is the use of probability to evaluate and choose among competing models, by comparing their marginal likelihoods or posterior probabilities, by estimating their expected predictive accuracy, or by averaging over them in proportion to their support.

Scope

This area covers Bayes factors and the marginal likelihood, predictive information criteria such as WAIC and leave-one-out cross-validation, Bayesian model averaging that accounts for model uncertainty, and posterior predictive checking for assessing absolute model fit.

Sub-topics

• bayes factors and marginal likelihood
• bayesian model averaging
• posterior predictive checks
• predictive information criteria

Core questions

• How do Bayes factors and posterior model probabilities compare models?
• How is expected predictive accuracy estimated using WAIC and cross-validation?
• How does Bayesian model averaging handle uncertainty about which model is correct?
• How do posterior predictive checks assess whether a single model fits the data?

Key concepts

• Bayes factor
• marginal likelihood
• WAIC
• leave-one-out cross-validation
• Bayesian model averaging
• posterior predictive check
• Occam's razor
• predictive accuracy

Key theories

Bayes factors

The ratio of marginal likelihoods quantifies the evidence the data provide for one model over another and is the formal Bayesian basis for hypothesis and model comparison.

Predictive model evaluation

Information criteria such as WAIC and efficient leave-one-out cross-validation estimate out-of-sample predictive accuracy directly from posterior draws, providing a prediction-focused alternative to Bayes factors.

Clinical relevance

Model comparison guides which scientific or predictive model to trust in fields from genetics to cosmology, and posterior predictive checks provide a principled way to detect model misfit before conclusions are drawn.

History

Jeffreys introduced Bayes factors for hypothesis testing in the 1930s; Kass and Raftery's 1995 review made them widely accessible. Concern about the marginal likelihood's sensitivity to priors and computation spurred predictive criteria such as DIC, WAIC, and efficient leave-one-out cross-validation.

Debates

Bayes factors versus predictive criteria

Bayes factors depend sensitively on the prior and can be hard to compute, while predictive criteria target out-of-sample accuracy; which to prefer depends on whether the goal is evidence for a hypothesis or predictive performance.

Key figures

• Harold Jeffreys
• Robert Kass
• Adrian Raftery
• Sumio Watanabe
• Aki Vehtari

Seminal works

• kass1995
• vehtari2017
• gelman2013

Frequently asked questions

Should I use Bayes factors or an information criterion?

Use Bayes factors when you want a measure of evidence for one hypothesis over another and can specify priors carefully; use predictive criteria such as WAIC or leave-one-out cross-validation when the goal is to compare expected out-of-sample predictive performance.