Unbiased Estimation and the Cramer-Rao Bound

Among estimators that are right on average, the Cramer-Rao inequality sets a floor on variance, and the Rao-Blackwell and Lehmann-Scheffe theorems show how to reach it.

Definition

An estimator is unbiased if its expected value equals the parameter for every parameter value; the Cramer-Rao bound states that the variance of any unbiased estimator is at least the inverse of the Fisher information.

Scope

This topic covers unbiasedness and its limitations, Fisher information for one and several parameters, the Cramer-Rao lower bound on the variance of an unbiased estimator, the conditions for attaining the bound, the Rao-Blackwell theorem on improving an estimator by conditioning on a sufficient statistic, and the Lehmann-Scheffe theorem identifying the unique minimum-variance unbiased estimator via complete sufficient statistics.

Core questions

• What is Fisher information, and how does it quantify the precision available in the data?
• Why can no unbiased estimator have variance below the Cramer-Rao bound, and when is the bound attained?
• How does conditioning on a sufficient statistic, via Rao-Blackwell, reduce variance?
• How do completeness and sufficiency together, via Lehmann-Scheffe, single out the best unbiased estimator?

Key theories

Cramer-Rao information inequality

Under regularity conditions the variance of an unbiased estimator is bounded below by the reciprocal of the Fisher information, defining efficiency as attainment of this bound.

Rao-Blackwell and Lehmann-Scheffe theorems

Conditioning any unbiased estimator on a sufficient statistic never increases its variance; if that statistic is also complete, the result is the unique minimum-variance unbiased estimator.

Clinical relevance

The Cramer-Rao bound and Fisher information set the fundamental precision limit of an experiment, guiding optimal experimental design and sensor calibration, while minimum-variance unbiased estimators provide benchmark estimates against which practical procedures are compared.

History

Cramer and Rao independently established the variance bound around 1945. Rao and Blackwell's improvement-by-conditioning result and Lehmann and Scheffe's uniqueness theorem followed in the late 1940s and early 1950s, completing the classical theory of unbiased estimation.

Key figures

• Calyampudi Radhakrishna Rao
• Harald Cramer
• David Blackwell
• Henry Scheffe

Seminal works

• lehmannCasella1998

Frequently asked questions

Is the Cramer-Rao bound always achievable?

No. It is attained only in special cases, mainly exponential families; in general the minimum-variance unbiased estimator may have variance strictly above the bound.

What does Fisher information measure?

It measures how sharply the likelihood responds to changes in the parameter, and hence how much information the data carry about it; larger Fisher information allows more precise estimation.