Stationary Distributions and Ergodicity
A stationary distribution is a probability distribution over states that a Markov chain leaves unchanged, and under mild conditions the chain forgets its starting point and converges to this equilibrium, with time averages matching space averages.
Definition
A stationary distribution of a Markov chain is a probability distribution over the states that is invariant under one step of the chain, and a chain is ergodic when, from any starting state, its distribution converges to this stationary distribution and its time averages converge to stationary expectations.
Scope
The topic covers stationary and invariant distributions and their existence and uniqueness for irreducible positive-recurrent chains, the role of aperiodicity in convergence, detailed balance and reversibility, the Markov chain ergodic theorem equating long-run time averages with stationary expectations, the rate of convergence to equilibrium and mixing times, and the use of these ideas in Markov chain Monte Carlo.
Core questions
- When does a Markov chain possess a unique stationary distribution?
- Under what conditions does the chain's distribution converge to that stationary distribution?
- What is detailed balance, and how does reversibility simplify finding the stationary distribution?
- How do long-run time averages relate to averages under the stationary distribution?
Key concepts
- stationary distribution
- irreducibility and aperiodicity
- detailed balance
- ergodic theorem
- mixing time
Key theories
- Existence, uniqueness, and convergence to stationarity
- An irreducible positive-recurrent Markov chain has a unique stationary distribution given by the reciprocals of mean return times, and if it is also aperiodic the distribution of the state converges to it from every starting point.
- Markov chain ergodic theorem
- For an irreducible positive-recurrent chain the long-run average of a function of the state converges almost surely to its expectation under the stationary distribution, the analogue of the law of large numbers for dependent Markov data.
- Detailed balance and reversibility
- If a distribution satisfies detailed balance with the transition probabilities, meaning the flow between any two states balances in both directions, then it is stationary and the chain is reversible, a condition exploited to design Markov chain Monte Carlo samplers.
Clinical relevance
These results are the theoretical engine of Markov chain Monte Carlo, where a chain is designed to have a target distribution as its stationary law so that its samples approximate that distribution; mixing-time bounds tell practitioners how long to run such simulations, and the same theory governs equilibrium queue lengths and steady-state reliability.
History
The equilibrium theory of Markov chains grew from Markov's original work and was placed in its modern form by Doob, Feller, and others. Its applied importance surged with the Metropolis algorithm of 1953 and Hastings' generalization of 1970, which turned convergence to a stationary distribution into a practical method of computation.
Key figures
- Andrey Markov
- Nicholas Metropolis
- Wilfred Keith Hastings
- Sean Meyn
Related topics
Seminal works
- norris1997
Frequently asked questions
- Does every Markov chain converge to a stationary distribution?
- No; convergence requires conditions such as irreducibility, positive recurrence, and aperiodicity. A periodic chain may cycle without settling, and a transient or null-recurrent chain may have no stationary distribution at all.
- Why is reversibility useful in practice?
- Reversibility via detailed balance gives a simple equation that a candidate stationary distribution must satisfy, which both makes the stationary distribution easy to verify and provides the design principle behind Metropolis-Hastings and many other Markov chain Monte Carlo algorithms.