Empirical Bayes Methods

Empirical Bayes estimates the prior distribution from the data themselves, providing much of the benefit of a hierarchical model at lower computational cost.

Definition

Empirical Bayes is an approach to hierarchical inference in which the parameters of the prior are estimated from the observed data, typically by maximizing the marginal likelihood, and then treated as known when computing posteriors for the group-level quantities.

Scope

This topic covers parametric and nonparametric empirical Bayes, estimation of hyperparameters by marginal maximum likelihood or method of moments, the connection to James-Stein shrinkage, and the caveat that empirical Bayes can understate uncertainty by ignoring error in the estimated prior.

Core questions

• How are hyperparameters estimated from the marginal distribution of the data?
• How does empirical Bayes relate to fully Bayesian hierarchical modeling?
• Why does it connect to James-Stein shrinkage estimators?
• In what way can empirical Bayes underestimate uncertainty?

Key concepts

• empirical Bayes
• marginal maximum likelihood
• hyperparameter estimation
• James-Stein estimator
• shrinkage
• false discovery rate
• uncertainty underestimation

Key theories

Estimating the prior from data

By fitting the prior's hyperparameters to the marginal distribution of all the data, empirical Bayes learns how much to pool without specifying a hyperprior, approximating the full hierarchical posterior.

Connection to Stein shrinkage

The James-Stein estimator can be derived as a parametric empirical Bayes rule, making explicit that data-estimated priors produce the shrinkage that reduces total error.

Clinical relevance

Empirical Bayes underlies large-scale inference in genomics and imaging, where thousands of effects are estimated simultaneously and data-driven priors stabilize the estimates and control false discoveries.

History

Robbins introduced empirical Bayes in 1956; Efron and Morris connected it to Stein shrinkage in the 1970s. The rise of high-throughput data made empirical Bayes central to large-scale simultaneous inference, as developed in Efron's 2010 monograph.

Debates

Ignoring uncertainty in the estimated prior

Because empirical Bayes plugs in point estimates of the hyperparameters, it can produce overconfident intervals relative to a fully Bayesian analysis that propagates that uncertainty.

Key figures

• Herbert Robbins
• Bradley Efron
• Carl Morris

Seminal works

• robbins1956
• efron2010

Frequently asked questions

Is empirical Bayes really Bayesian?

It is a hybrid: it uses Bayes' theorem for the group-level parameters but estimates the prior from the data rather than specifying it in advance, which approximates a full hierarchical model while typically understating uncertainty in the prior.