P-Value and Statistical Significance

It means that if the null hypothesis were true, the probability of observing data as extreme as or more extreme than what you obtained is 0.03 (3%). It does NOT mean there is a 3% probability the null is true, or a 97% probability the alternative is true. Those are common misconceptions that conflate p-values with posterior probabilities (which require Bayesian inference).

Fisher chose 0.05 somewhat arbitrarily as a convenient threshold in 1925, balancing the desire to detect real effects with the willingness to tolerate false positives. It has become convention by decades of use, but there is no statistical reason it must be 0.05. Some fields use 0.01 for high-stakes decisions, and pre-registration allows you to choose any α a priori.

Under the strict decision rule α = 0.05, yes—you would fail to reject H₀. However, p = 0.051 and p = 0.049 carry virtually the same evidence. The binary threshold creates a false cliff. This is why researchers increasingly report the actual p-value and effect size rather than simply declaring 'significant' or 'not significant.'

Not as hypothesis tests. In exploratory work, p-values from multiple comparisons without correction will be inflated false positives. If you conduct exploratory analyses, report them as exploratory, adjust p-values for multiple comparisons (Bonferroni, FDR), or use them only to generate hypotheses for future confirmatory studies. Pre-registration separates exploratory from confirmatory work.

No. Failure to reject H₀ does not mean H₀ is true; it means the data do not provide sufficient evidence to reject it. You may have insufficient power (Type II error), confounding variables, or measurement error. Always report confidence intervals and effect sizes alongside non-significant results.