Resampling Methods

Resampling methods assess the uncertainty of a statistic by repeatedly drawing new samples from the observed data, replacing analytic formulas for standard errors and distributions with computation.

Definition

Resampling methods are computer-intensive inferential techniques that estimate the sampling distribution, bias, variance, or predictive error of a statistic by repeatedly recomputing it on samples drawn from, or partitions of, the observed data.

Scope

This area covers the bootstrap and its confidence intervals, the jackknife for bias and variance estimation, permutation and randomization tests for hypothesis testing, and cross-validation for estimating predictive error. The unifying idea is that the empirical distribution of the data, reused through resampling, substitutes for an unknown population distribution.

Sub-topics

• bootstrap methods
• cross validation resampling
• jackknife resampling
• permutation tests

Core questions

• How can repeatedly resampling the observed data approximate the sampling distribution of a statistic?
• What distinguishes the bootstrap, the jackknife, permutation tests, and cross-validation in aim and mechanism?
• When do resampling approximations succeed, and where do they break down?
• How are resampling methods used to build confidence intervals and tests without parametric assumptions?

Key theories

The plug-in principle

Resampling replaces the unknown population distribution with the empirical distribution of the sample, so that quantities such as standard errors and biases are computed by repeated sampling from the data itself.

Resampling for inference

Bootstrap resampling estimates variability and confidence intervals, permutation resampling generates exact or approximate null distributions, and cross-validation reuses partitions of the data to estimate out-of-sample error.

Clinical relevance

Resampling methods deliver standard errors, confidence intervals and tests for complicated statistics where no tractable formula exists, and provide honest estimates of predictive accuracy for statistical and machine-learning models; their minimal assumptions make them ubiquitous across the empirical sciences.

History

Quenouille and Tukey developed the jackknife in the 1940s and 1950s; Efron introduced the bootstrap in 1979 and unified it with the jackknife, and the rise of cheap computing through the 1980s and 1990s made resampling a mainstream alternative to asymptotic theory.

Key figures

• Bradley Efron
• Robert Tibshirani
• Anthony Davison
• Maurice Quenouille

Seminal works

• efron1993
• efron1979

Frequently asked questions

Why are resampling methods called computer-intensive?

They replace closed-form derivations with thousands of repeated recomputations of a statistic on resampled data. This is only practical with a computer, but in exchange the methods require far fewer modeling assumptions.

Do resampling methods always work?

No. They can fail for statistics that depend on extreme order statistics, for very small samples, or under strong dependence. Knowing these failure modes is part of using the methods responsibly.