Model Confidence Set (MCS)
Also known as: MCS Procedure, Superior Set of Models, Model Selection Confidence Set, Model Güven Kümesi
The Model Confidence Set (MCS) is a sequential hypothesis-testing procedure introduced by Hansen, Lunde, and Nason (2011) that identifies the smallest collection of forecasting or predictive models statistically indistinguishable from the best-performing model at a given confidence level. Instead of selecting a single winner, MCS returns a set of superior models, making it especially valuable in econometric forecast comparisons where the true best model is unknown.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use MCS when comparing multiple competing forecasting or regression models and you want a statistically rigorous, uncertainty-aware shortlist rather than a single champion. It is appropriate when the number of models is moderate (up to several dozen), a common evaluation sample exists, and loss differentials may exhibit serial correlation. Key assumptions include covariance stationarity of loss differentials and sufficient sample length for reliable bootstrap inference. MCS is less suitable with very small T, non-stationary losses, or when a single model selection is mandated. Alternatives include the Diebold-Mariano test for pairwise comparison or the Giacomini-White test for conditional predictive ability.
Strengths & limitations
- Produces a confidence set rather than a fragile point selection, honestly representing data-limited discrimination
- Controls familywise error rate across multiple sequential comparisons via bootstrap
- Distribution-free: works with any loss function (MSE, MAE, QLIKE, etc.) without assuming normality
- Directly applicable to nested and non-nested model comparisons
- Requires a sufficiently long and common evaluation sample; short T inflates bootstrap variance and yields wide sets
- Power decreases as the number of candidate models grows, potentially retaining many near-equivalent models
- Block bootstrap performance depends on correct block-length selection, which is data-dependent
- Does not rank models within the set; all surviving models are treated as statistically equivalent
Frequently asked
How does MCS differ from the Diebold-Mariano test?
The Diebold-Mariano test is a pairwise comparison between exactly two models, providing a binary reject/not-reject decision. MCS generalizes this to multiple models simultaneously, using sequential elimination and familywise error control, and returns a set of indistinguishable superior models rather than declaring a single winner.
What confidence level should I choose for the MCS?
Hansen et al. (2011) recommend α = 0.10 or α = 0.25 in empirical applications because conventional α = 0.05 often yields overly restrictive sets with short samples. The choice should reflect the research context: a smaller α is more conservative and produces larger sets, while a larger α is more selective but risks excluding genuinely competitive models.
Can MCS be applied to in-sample model fit comparisons?
MCS is designed for out-of-sample forecast evaluation where loss differentials are computed over a hold-out period. Applying it to in-sample criteria such as AIC or BIC is non-standard and not recommended, because the stationarity and independence assumptions underlying the bootstrap critical values are more naturally satisfied in genuine out-of-sample settings.
Sources
- Hansen, P. R., Lunde, A., & Nason, J. M. (2011). The model confidence set. Econometrica, 79(2), 453–497. DOI: 10.2139/ssrn.522382 ↗
How to cite this page
ScholarGate. (2026, June 2). Model Confidence Set (MCS). ScholarGate. https://scholargate.app/en/econometrics/model-confidence-set
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Diebold-Mariano TestEconometrics↔ compare
- Giacomini-White TestEconometrics↔ compare
- Stepwise RegressionStatistics↔ compare