Predictive Information Criteria

Predictive information criteria estimate a model's expected out-of-sample accuracy from its posterior, offering a prediction-focused alternative to Bayes factors for comparing models.

Definition

Predictive information criteria are estimates of a model's expected log predictive density on new data, computed from posterior samples and corrected for overfitting by an effective-parameter penalty, used to rank models by predictive performance.

Scope

This topic covers the deviance information criterion (DIC), the widely applicable information criterion (WAIC), and efficient Pareto-smoothed importance-sampling leave-one-out cross-validation, including how each estimates the effective number of parameters and approximates expected log predictive density.

Core questions

• How do DIC, WAIC, and leave-one-out cross-validation estimate predictive accuracy?
• What is the effective number of parameters and how is it computed?
• Why is WAIC considered more fully Bayesian than DIC?
• How does Pareto-smoothed importance sampling make leave-one-out cross-validation efficient?

Key concepts

• DIC
• WAIC
• leave-one-out cross-validation
• expected log predictive density
• effective number of parameters
• Pareto-smoothed importance sampling
• overfitting penalty

Key theories

Effective number of parameters

Each criterion penalizes fit by an estimate of model complexity derived from the variability of the log-likelihood across the posterior, so that better in-sample fit does not automatically win.

WAIC and cross-validation equivalence

Watanabe showed that WAIC is asymptotically equivalent to Bayesian leave-one-out cross-validation, and both directly target expected out-of-sample log predictive density using the full posterior.

Clinical relevance

Predictive criteria let researchers compare candidate models for prediction in epidemiology, ecology, and the physical sciences without specifying the carefully tuned priors that Bayes factors demand.

History

Spiegelhalter and colleagues proposed DIC in 2002; Watanabe introduced WAIC from singular learning theory in 2010. Vehtari, Gelman, and Gabry's 2017 work on Pareto-smoothed importance-sampling leave-one-out cross-validation made stable, diagnosable predictive evaluation practical.

Debates

Reliability of DIC

DIC can behave poorly for hierarchical and non-regular models and lacks invariance, leading many to prefer WAIC or leave-one-out cross-validation, though no single criterion is universally best.

Key figures

• David Spiegelhalter
• Sumio Watanabe
• Aki Vehtari
• Andrew Gelman

Seminal works

• watanabe2010
• vehtari2017

Frequently asked questions

Is a lower or higher information criterion better?

These criteria are usually reported on a deviance scale where lower values indicate better estimated out-of-sample predictive accuracy; differences should be judged relative to their standard errors rather than treated as exact.