Laws of Large Numbers

The laws of large numbers state that the average of many independent observations of a random quantity converges to its expected value, giving mathematical content to the intuition that long-run frequencies stabilize.

Definition

The laws of large numbers assert that the sample average of independent identically distributed random variables with a finite mean converges to that mean, in probability for the weak law and almost surely for the strong law.

Scope

The topic covers the weak law of large numbers proved by Chebyshev's inequality and by truncation, Khinchin's weak law under only a finite mean, Kolmogorov's strong law of large numbers with its maximal inequality and three-series theorem, the distinction between convergence in probability and almost-sure convergence, and the failure of the laws for variables without a finite mean.

Core questions

• In what precise sense does a sample average approach the true mean as the sample grows?
• What is the difference between the weak and strong laws, and what hypotheses does each need?
• Which inequalities and decompositions make the strong law provable?
• What happens when the underlying distribution has no finite mean?

Key concepts

• convergence in probability
• almost-sure convergence
• Chebyshev inequality
• truncation method
• Kolmogorov three-series theorem

Key theories

Weak law of large numbers

For independent identically distributed variables with finite mean the sample average converges to the mean in probability, a result obtainable from Chebyshev's inequality when the variance is finite and from truncation arguments under Khinchin's weaker hypothesis.

Kolmogorov strong law of large numbers

For independent identically distributed variables a finite mean is necessary and sufficient for the sample average to converge to the mean almost surely, the definitive form of the law and the basis for the frequency interpretation of probability.

Clinical relevance

The strong law is what licenses estimating an expectation by a sample mean and underlies Monte Carlo integration, the consistency of estimators in statistics, and the frequentist interpretation of probability as long-run relative frequency; its failure for heavy-tailed data warns against averaging quantities with infinite mean such as certain insurance losses.

History

Bernoulli proved the first law of large numbers for binomial proportions in 1713. Chebyshev gave a simple variance-based proof, Khinchin weakened the hypotheses to a finite mean, and Kolmogorov established the definitive almost-sure strong law together with the maximal inequality and three-series theorem that prove it.

Key figures

• Jacob Bernoulli
• Pafnuty Chebyshev
• Aleksandr Khinchin
• Andrey Kolmogorov

Seminal works

• billingsley1995

Frequently asked questions

What is the difference between the weak and strong laws of large numbers?

The weak law says the average is likely to be close to the mean for any large fixed sample size, whereas the strong law says that with probability one the entire sequence of averages converges to the mean; the strong law is the more definitive statement.

Can the law of large numbers fail?

Yes; if the underlying distribution has no finite mean, such as the Cauchy distribution, the sample average does not converge to a constant at all, and the law in its usual form does not apply.