Hypothesis Testing Framework

The hypothesis testing framework is a structured procedure for deciding whether sample data are compatible with a specified claim about a population. It pits a null hypothesis - usually a statement of no effect or no difference - against an alternative, computes a test statistic and an associated p-value, and uses a pre-set significance level to judge whether the evidence against the null is strong enough to act on. It is the most widely used, and most widely debated, decision procedure in quantitative health research.

Definition

Statistical hypothesis testing is a procedure that uses a test statistic computed from sample data to assess the compatibility of the data with a stated null hypothesis, rejecting the null in favour of an alternative when the result is sufficiently unlikely under the null at a pre-specified significance level.

Scope

This topic covers the logic of the null and alternative hypotheses, the role of the test statistic and p-value, the meaning of the significance level, and the major criticisms of mechanical significance testing. It is a reference methodology for designing and appraising studies, not a clinical decision rule.

Core questions

• What null hypothesis is being tested, and against what alternative?
• How surprising are the observed data if the null hypothesis were true?
• What significance level governs the decision, and why?
• What does rejecting - or failing to reject - the null actually license us to conclude?

Key concepts

• Null hypothesis
• Alternative hypothesis
• Test statistic
• P value
• Significance level (alpha)
• Rejection region
• One- and two-sided tests
• Statistical versus practical significance

Key theories

Neyman-Pearson testing

Casts testing as a choice between two hypotheses with controlled long-run error rates, defining the significance level (Type I error rate) in advance and seeking the test that maximises power against the alternative.

Mechanisms

A test begins by stating a null hypothesis and an alternative, then summarising the data in a test statistic whose distribution under the null is known. The p-value is the probability, computed under the null, of a result at least as extreme as the one observed; a small p-value indicates that the data would be unusual if the null were true. If the p-value falls below the pre-specified significance level, the null is rejected. Crucially, the p-value is not the probability that the null is true, and failing to reject is not proof that the null is correct. Statistical significance also need not imply a meaningful effect, which is why the framework is read alongside effect estimates and confidence intervals.

Clinical relevance

Hypothesis tests underlie the headline conclusions of most trials and observational studies, so understanding what a significant or non-significant result does and does not mean is central to evidence appraisal. Misreading a p-value can lead to over- or under-stating findings. This entry describes the inferential procedure and is not a basis for individual diagnostic or treatment decisions.

Evidence & guidelines

Concern about mechanical use of significance thresholds prompted the American Statistical Association's 2016 statement clarifying the proper interpretation of p-values, and Greenland and colleagues' guide to common misinterpretations. Some methodologists have proposed lowering the conventional threshold for claims of new discoveries, while others argue against any fixed threshold - debates that remain unresolved.

History

The framework fuses Fisher's significance testing and p-values with the decision-theoretic testing of Neyman and Pearson, formalised in 1933, which introduced fixed error rates and the most powerful test. The hybrid that became standard textbook practice drew criticism throughout the twentieth century, intensifying in the 2010s amid concerns about reproducibility, leading to formal cautionary statements and proposals to redefine or abandon fixed significance thresholds.

Debates

The status of the 0.05 significance threshold

Critics argue that a fixed conventional threshold encourages dichotomous, sometimes misleading conclusions; proposals range from lowering the threshold for new claims to abandoning bright-line thresholds in favour of continuous interpretation of evidence.

Key figures

• Ronald A. Fisher
• Jerzy Neyman
• Egon Pearson
• Sander Greenland
• Ronald L. Wasserstein

Seminal works

• neyman-pearson-1933
• wasserstein-lazar-2016

Frequently asked questions

Does a p-value tell me the probability that the null hypothesis is true?

No. The p-value is the probability of data at least as extreme as observed assuming the null hypothesis is true; it does not give the probability that the null itself is true or false.

Does failing to reject the null hypothesis prove there is no effect?

No. A non-significant result means the data did not provide enough evidence against the null, which can occur simply because the study was too small; absence of evidence is not evidence of absence.