Metropolis-Hastings for Model Comparison
Metropolis-Hastings Algorithm for Bayesian Model Comparison · Also known as: MH model comparison, Metropolis-Hastings Bayes factor estimation, reversible-jump Metropolis-Hastings, MH model selection
Metropolis-Hastings for model comparison uses the Metropolis-Hastings MCMC algorithm to explore both parameter and model space simultaneously, producing posterior probabilities for competing models and enabling Bayes factor estimation without requiring closed-form marginal likelihoods. The canonical extension — reversible-jump MCMC by Green (1995) — handles models of different dimensionalities within a single sampler.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use Metropolis-Hastings for model comparison when you need to compare models of potentially different dimensionality within a single MCMC run, when computing the marginal likelihood analytically is infeasible, or when you want posterior model probabilities and model-averaged parameter estimates together. It is especially suited to variable selection (regression with an unknown subset of predictors) and mixture order selection. Do not use it when models are nested and a likelihood-ratio test suffices, when the number of models is so large that no sampler can adequately explore model space, or when marginal likelihoods can be computed in closed form (e.g., conjugate models where Bayes factors are available directly).
Strengths & limitations
- Explores both parameter and model uncertainty in a single MCMC run, yielding coherent Bayesian inference.
- Handles models of different dimensionalities via reversible-jump moves, requiring no common parametrisation.
- Produces posterior model probabilities directly, enabling Bayes factors and model-averaged predictions.
- Well-understood theoretical guarantees: the chain converges to the correct joint posterior over models and parameters.
- Applicable to virtually any likelihood and prior without closed-form marginal likelihoods.
- Designing efficient between-model proposal distributions is non-trivial and strongly problem-dependent.
- Poor between-model proposals lead to very low inter-model acceptance rates and chains that get trapped in one model.
- Computational cost grows with the number of models and their complexity, making exhaustive model search infeasible.
- Posterior model probabilities are sensitive to prior specification, especially for non-nested or large models.
Frequently asked
What is reversible-jump MCMC and why is it needed?
Standard Metropolis-Hastings moves within a fixed-dimensional parameter space. When comparing models with different numbers of parameters, a dimension-matching step must be introduced so that the joint proposal map is invertible. Green's 1995 reversible-jump construction adds auxiliary random variables to equalise dimensions and includes a Jacobian correction in the acceptance ratio to maintain detailed balance.
How does this approach compare with computing Bayes factors directly?
Direct Bayes factor computation requires the marginal likelihood of each model, which often has no closed form and requires its own numerical integration (bridge sampling, thermodynamic integration, etc.). Reversible-jump MCMC estimates posterior model probabilities — and hence Bayes factors — as a by-product of a single run, but its accuracy depends on how well the chain mixes across models.
How do I know if the chain is mixing well across models?
Monitor the inter-model acceptance rate (typically target 20-40% for between-model moves), inspect the model-index trace plot to check that all models are visited repeatedly, and compute the effective sample size for the model indicator. If the chain rarely leaves a dominant model, the between-model proposals need re-tuning.
Can I use this for comparing many models simultaneously?
In principle yes, but in practice the sampler struggles when the number of candidate models is exponentially large (e.g., variable selection with many predictors). Stochastic search variable selection (SSVS) or continuous spike-and-slab priors may be more practical alternatives for high-dimensional model spaces.
Are the posterior model probabilities sensitive to the model prior?
Yes, significantly. Posterior model probabilities are always proportional to the prior probability of each model times its marginal likelihood. With few data or models of similar fit, the prior dominates. Always report results under at least two different model priors and discuss sensitivity.
Sources
- Hastings, W. K. (1970). Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1), 97-109. DOI: 10.1093/biomet/57.1.97 ↗
- Green, P. J. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika, 82(4), 711-732. DOI: 10.1093/biomet/82.4.711 ↗
How to cite this page
ScholarGate. (2026, June 3). Metropolis-Hastings Algorithm for Bayesian Model Comparison. ScholarGate. https://scholargate.app/en/bayesian/metropolis-hastings-for-model-comparison
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Bayesian Model AveragingBayesian↔ compare
- Gibbs Sampling for Model ComparisonBayesian↔ compare
- MCMC for Model ComparisonBayesian↔ compare
- Sequential Monte CarloBayesian↔ compare