What is the difference between the law of large numbers and the central limit theorem?

The law of large numbers says the average converges to the mean, describing first-order behavior, while the central limit theorem describes the second-order fluctuations of the average around the mean, which are Gaussian on the scale of one over the square root of the sample size.

Does the central limit theorem always apply?

It requires conditions such as finite variance and a negligibility condition like Lindeberg's; for heavy-tailed variables with infinite variance the limit can instead be a non-Gaussian stable distribution.

Limit Theorems

Limit theorems describe what happens to sums and averages of many random variables: they stabilize around their mean by the laws of large numbers, fluctuate on a fine scale according to the central limit theorem, and deviate by large amounts only with exponentially small probability.

Definition

Limit theorems are the body of results describing the asymptotic behavior of sequences of random variables and their distributions, principally the convergence of averages to expectations, the Gaussian fluctuations of normalized sums, and the exponential decay of large-deviation probabilities.

Scope

The area covers the weak and strong laws of large numbers, the classical and Lindeberg-Feller central limit theorems with their characteristic-function proofs, the hierarchy of convergence modes for random variables and distributions, weak convergence of probability measures with tightness, and the theory of large deviations governing exponentially rare events.

Sub-topics

Core questions

In what senses does the average of many random variables converge to its mean?
Why are the fluctuations of a normalized sum approximately Gaussian under broad conditions?
How are the different modes of convergence for random variables and distributions related?
How rare are large deviations from the typical behavior, and at what rate do they decay?

Key theories

Laws of large numbers: The averages of independent identically distributed variables with finite mean converge to that mean, in probability for the weak law and almost surely for the strong law, which is the mathematical justification for estimating expectations by sample averages.
Central limit theorem: Sums of independent variables with finite variance, suitably centered and scaled, converge in distribution to a normal law, explaining the ubiquity of the Gaussian and providing the basis for confidence intervals and significance tests.

Clinical relevance

Limit theorems are the theoretical guarantee behind statistical practice and simulation: the law of large numbers validates Monte Carlo estimation and frequentist interpretation of probability, the central limit theorem justifies normal-based inference and many approximate methods, and large-deviation rates quantify rare-event risk in insurance, communications, and reliability.

History

The first limit theorem was Bernoulli's law of large numbers; de Moivre and Laplace found the normal approximation to the binomial, generalized by Lyapunov and Lindeberg into the central limit theorem. Kolmogorov sharpened the strong law, Cramer founded large-deviation theory, and the modern measure-theoretic treatment unifies them.

Key figures

Jacob Bernoulli
Aleksandr Lyapunov
Paul Levy
Harald Cramer

Seminal works

billingsley1995
billingsley1999convergence

Frequently asked questions

What is the difference between the law of large numbers and the central limit theorem?: The law of large numbers says the average converges to the mean, describing first-order behavior, while the central limit theorem describes the second-order fluctuations of the average around the mean, which are Gaussian on the scale of one over the square root of the sample size.
Does the central limit theorem always apply?: It requires conditions such as finite variance and a negligibility condition like Lindeberg's; for heavy-tailed variables with infinite variance the limit can instead be a non-Gaussian stable distribution.