How is a permutation test different from a bootstrap test?

A permutation test resamples by rearranging labels without replacement under a null of exchangeability, giving an exact test of that null. The bootstrap resamples with replacement to approximate a sampling distribution and is aimed primarily at estimating uncertainty rather than testing exchangeability.

When can a permutation test be exact?

When all relevant permutations can be enumerated and the exchangeability assumption holds, the resulting p-value is exact. For large samples the permutations are sampled at random instead, giving an arbitrarily accurate approximation.

Permutation Tests

A permutation test assesses a hypothesis by comparing an observed statistic to the distribution of that statistic obtained from all relabellings of the data that the null hypothesis treats as exchangeable.

Thema finden mit PaperMindDemnächstFind papers & topics

Tools & resources

Folien herunterladen

Learn & explore

VideoDemnächst

Definition

A permutation test is a nonparametric hypothesis test that builds the null distribution of a test statistic by recomputing it over the rearrangements of the data that are equally likely under a null hypothesis of exchangeability, then locating the observed value within that distribution.

Scope

This topic covers the exchangeability assumption that justifies permutation inference, the construction of the permutation null distribution, exact versus Monte Carlo approximations when the number of permutations is too large to enumerate, two-sample and paired designs, and the relationship of permutation tests to classical and bootstrap procedures.

Core questions

What exchangeability assumption makes the permutation null distribution valid?
How is the permutation distribution of a statistic constructed and used to obtain a p-value?
When must the permutation distribution be approximated by random sampling rather than full enumeration?
How do permutation tests relate to classical parametric tests and to the bootstrap?

Key concepts

Exchangeability
Permutation null distribution
Monte Carlo p-value
Test statistic
Randomization inference

Key theories

Exchangeability and the permutation null: If observations are exchangeable under the null hypothesis, every relabelling is equally likely, so the test statistic's distribution over all permutations is its exact null distribution, yielding a test with exact level.
Monte Carlo permutation: When the number of permutations is astronomically large, a random sample of permutations approximates the null distribution, giving a Monte Carlo p-value whose accuracy is controlled by the number of permutations drawn.

Clinical relevance

Permutation tests provide assumption-light, often exact hypothesis tests for two-sample comparisons, paired designs and complex statistics, and they underpin multiple-testing procedures and significance assessment in genomics, neuroimaging and randomized experiments.

History

Fisher and Pitman formulated permutation (randomization) tests in the 1930s as the exact justification for analyses of designed experiments; computationally infeasible at the time, they became practical once computers could enumerate or sample large permutation sets.

Key figures

Ronald A. Fisher
Edwin Pitman
Phillip Good

Seminal works

good2005
davison1997

Frequently asked questions

How is a permutation test different from a bootstrap test?: A permutation test resamples by rearranging labels without replacement under a null of exchangeability, giving an exact test of that null. The bootstrap resamples with replacement to approximate a sampling distribution and is aimed primarily at estimating uncertainty rather than testing exchangeability.
When can a permutation test be exact?: When all relevant permutations can be enumerated and the exchangeability assumption holds, the resulting p-value is exact. For large samples the permutations are sampled at random instead, giving an arbitrarily accurate approximation.