Permutation (Randomization) Test
Also known as: randomization test, exact permutation test, re-randomization test, Permütasyon Testi
The permutation test is a nonparametric resampling procedure that builds the sampling distribution of a test statistic directly from the data by repeatedly shuffling the group labels. Developed in the resampling tradition and treated systematically by Good (2005) and Edgington & Onghena (2007), it requires no parametric distributional assumption and yields an exact p-value.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+17 more
When to use it
Use a permutation test to compare groups or assess a relationship when parametric assumptions such as normality are doubtful, when the sample is moderate, or when you want an exact p-value free of distributional approximation. It applies to continuous, categorical, and ordinal data and to a wide range of designs. The key requirement is that observations be exchangeable under the null hypothesis, and you should run enough permutations (at least about 1000) for a stable result. Avoid it for very small samples (below about 10, and especially below 5), where the number of distinct permutations is too limited to give meaningful resolution.
Strengths & limitations
- Distribution-free: requires no normality or other parametric assumption, relying only on exchangeability under the null.
- Delivers an exact (or arbitrarily precise) p-value built directly from the data.
- Flexible — works with continuous, categorical, or ordinal data and with almost any test statistic you can compute.
- With fewer than about 10 observations the number of distinct permutations is limited, so an exact enumeration test is preferable.
- With fewer than 5 observations the permutation distribution is so restricted that results become meaningless.
- Validity hinges on exchangeability; structured dependence (clustering, time order) can break that assumption.
- Can be computationally heavy because the statistic is recomputed over many resamples.
Frequently asked
How many permutations do I need?
Run at least about 1000 random permutations for a stable result; more permutations sharpen the resolution of the p-value. For very small samples it can be feasible to enumerate every possible relabelling, giving a truly exact test.
What is exchangeability and why does it matter?
Exchangeability means that under the null hypothesis any reassignment of observations to groups is equally likely. It is the single assumption the permutation test relies on; if the data are clustered or ordered in time so that labels are not freely interchangeable, the test can be invalid.
How is a permutation test different from a bootstrap?
A permutation test reshuffles labels without replacement to test a null of no group difference, producing an exact p-value. The bootstrap resamples observations with replacement to estimate the variability of a statistic; it is aimed at confidence intervals and standard errors rather than an exact null test.
What should I do when the sample is too small?
Below about 10 observations the permutation distribution is limited, and below 5 it is essentially meaningless. In those cases prefer an exact enumeration test, or fall back to a jackknife or bootstrap inference procedure.
Sources
- Good, P. (2005). Permutation, Parametric and Bootstrap Tests of Hypotheses (3rd ed.). Springer. ISBN: 978-0387202792
- Edgington, E. S., & Onghena, P. (2007). Randomization Tests (4th ed.). CRC Press. ISBN: 978-1584885894
How to cite this page
ScholarGate. (2026, June 1). Permutation (Randomization) Test. ScholarGate. https://scholargate.app/en/statistics/permutation-test
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Bootstrap InferenceStatistics↔ compare
- JackknifeStatistics↔ compare
- Robust CorrelationStatistics↔ compare
- Trimmed Mean TestStatistics↔ compare
- Winsorized EstimationStatistics↔ compare