Bayes Factors and Marginal Likelihood

The marginal likelihood is the probability of the data under a model after integrating out its parameters, and the ratio of two marginal likelihoods, the Bayes factor, measures the evidence between models.

Definition

The marginal likelihood of a model is the integral of the likelihood over the prior; the Bayes factor between two models is the ratio of their marginal likelihoods and, multiplied by the prior odds, gives the posterior odds in favor of one model.

Scope

This topic covers the definition and interpretation of the marginal likelihood, the Bayes factor and its calibration into evidence categories, its automatic penalization of complexity, the Jeffreys-Lindley paradox showing sensitivity to diffuse priors, and computational methods such as bridge sampling.

Core questions

• What is the marginal likelihood and how does it embody an automatic Occam's razor?
• How is a Bayes factor interpreted as strength of evidence?
• Why are Bayes factors sensitive to the choice of prior, as shown by the Jeffreys-Lindley paradox?
• How is the marginal likelihood computed in practice?

Key concepts

• marginal likelihood
• Bayes factor
• posterior odds
• Occam's razor
• Jeffreys-Lindley paradox
• bridge sampling
• prior sensitivity

Key theories

Bayes factor as evidence

The Bayes factor converts prior odds into posterior odds and is read on calibrated scales as the weight of evidence the data give for one model over another.

Jeffreys-Lindley paradox

Because the marginal likelihood depends on the prior's spread, an arbitrarily diffuse prior can force the Bayes factor to favor the simpler model regardless of the data, so improper priors must not be used for model comparison.

Clinical relevance

Bayes factors provide a principled measure of evidence used in genetics, psychology, and physics for comparing hypotheses, but their dependence on the prior means they must be reported with the priors that produced them.

History

Jeffreys developed Bayes factors for hypothesis testing in the 1930s; Lindley's 1957 paradox exposed their sensitivity to diffuse priors. Kass and Raftery's 1995 review standardized their interpretation and surveyed computational approaches.

Debates

Use of improper or vague priors

Because the marginal likelihood is undefined for improper priors and unstable for very diffuse ones, there is debate about default priors for model comparison and whether Bayes factors are appropriate at all in such settings.

Key figures

• Harold Jeffreys
• Dennis Lindley
• Robert Kass
• Adrian Raftery

Seminal works

• kass1995
• lindley1957

Frequently asked questions

Can I use a noninformative prior to compute a Bayes factor?

Generally no: improper priors leave the marginal likelihood undefined and very diffuse proper priors bias the Bayes factor toward the simpler model, the essence of the Jeffreys-Lindley paradox, so Bayes factors require carefully chosen proper priors.