Multiple Comparisons Problem
The Multiple Comparisons Problem and Statistical Correction Methods · Also known as: multiple testing, family-wise error, p-value adjustment, false discovery rate
When conducting multiple statistical tests, the probability of obtaining at least one false positive by chance increases with the number of tests. The multiple comparisons problem (also called the multiplicity problem) occurs because if you conduct 100 hypothesis tests at α = 0.05, you expect ~5 false positives by chance alone, even if all null hypotheses are true. Correction methods—Bonferroni, Benjamini-Hochberg false discovery rate (FDR), and others—adjust the significance threshold or p-values to control error rates. This concept is critical for research integrity and has profound implications for exploratory science.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Always adjust for multiple comparisons when: (1) You test many hypotheses (e.g., >5). (2) You conduct subgroup analyses or sensitivity analyses. (3) You perform exploratory data analysis without pre-registration. Use Bonferroni when false positives are very costly (e.g., approving a drug, determining guilt in law). Use FDR when you can tolerate some false positives and can replicate findings (e.g., gene discovery, where replication identifies which findings are true). Do NOT correct if all tests are pre-specified as a single, unified hypothesis (e.g., testing 5 outcome variables as a multivariate composite). Do NOT over-correct: if you conduct only 1–2 tests, correction is unnecessary.
Strengths & limitations
- Prevents inflation of false positive rates in studies with many tests, protecting scientific integrity.
- Bonferroni is simple, transparent, and universally applicable to any type of test.
- FDR balances power and error control, enabling discovery in high-dimensional data (genomics, neuroimaging) while controlling false positives.
- Enables pre-registration of corrections; specifying correction method a priori reduces p-hacking.
- Bonferroni correction is conservative, especially with many tests; it may miss true effects (Type II error inflation).
- Assumes independence of tests; with dependent (correlated) tests, Bonferroni is overly conservative.
- FDR assumes you have enough tests and enough true signals to work reliably; with very few tests, FDR may not control Type I error well.
- Correction depends on counting total tests a priori; if you undercount or conduct unplanned tests post-hoc, correction is invalid.
- Does not protect against p-hacking or selective reporting; correction assumes all tests are reported, not just significant ones.
Frequently asked
Do I need to correct for multiple comparisons if I'm testing a multivariate hypothesis?
Not if you test a single composite hypothesis (e.g., testing whether a treatment affects any of 5 outcomes simultaneously using multivariate ANOVA). You are conducting one test. However, if you then follow up with univariate tests for each outcome, you should correct those follow-ups. Pre-specify whether your main hypothesis is multivariate (one test) or univariate multiple tests (needs correction).
Which is better: Bonferroni or FDR?
Depends on context. Bonferroni controls family-wise error (probability of ANY false positive), protecting against false positives strictly. FDR controls the proportion of false positives among discoveries, allowing some false positives but maintaining power. Use Bonferroni for high-stakes decisions (drug approval) where false positives are very costly. Use FDR for discovery research (genomics, imaging) where you can tolerate some false positives and can replicate findings.
If I conduct 100 exploratory tests and find 5 significant at p < 0.05, are these real findings?
Probably not—these 5 are likely false positives from random noise. If all 100 tests have true null hypotheses and α = 0.05, you expect 5 false positives. To claim these are real, either: (1) Apply multiple comparison correction (FDR or Bonferroni) a priori. (2) Replicate on a new, independent dataset. (3) Pre-register hypotheses before seeing data. Never report exploratory p-values as confirmatory evidence.
Can I apply correction only to some tests, not others?
Only if you pre-specify which tests are primary (uncorrected) and which are exploratory (corrected). Report this clearly. Post-hoc selective correction (correcting some but not other tests based on results) is p-hacking. The safest approach: pre-register all tests and correction method before data analysis.
What happens if I conduct an undisclosed number of tests?
You cannot know the correct correction factor. If you report only significant findings and hide the total number of tests conducted, readers cannot assess the false positive rate. This is selective reporting bias. Always disclose: (1) How many tests you planned (pre-registration). (2) How many you actually conducted. (3) What correction method you applied. Transparency enables readers to judge credibility.
Sources
- Bonferroni, C. E. (1935). Il calcolo dei coefficienti di correlazione nel caso di variabilità di gruppi. Instituto Italiano di Statistica. link ↗
- Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B, 57(1), 289–300. DOI: 10.1111/j.2517-6161.1995.tb02031.x ↗
- Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. DOI: 10.1371/journal.pmed.0020124 ↗
How to cite this page
ScholarGate. (2026, June 3). The Multiple Comparisons Problem and Statistical Correction Methods. ScholarGate. https://scholargate.app/en/research-statistics/multiple-comparisons-problem
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Null Hypothesis TestingResearch Statistics↔ compare
- P-Value and Statistical SignificanceResearch Statistics↔ compare
- Publication BiasResearch Statistics↔ compare
- Type I and Type II ErrorsResearch Statistics↔ compare