Maximum Likelihood Estimation

Ordinary least squares (OLS) minimises the sum of squared residuals and requires no distributional assumption beyond uncorrelated errors with constant variance. MLE maximises the likelihood given an explicit distributional assumption. Crucially, when the errors are normally distributed, OLS and MLE yield identical coefficient estimates for linear regression — but MLE additionally estimates the error variance and generalises naturally to non-normal outcomes (e.g., logistic or Poisson regression) where OLS is inappropriate.

The Fisher information I(θ) is the expected value of the squared score function, equivalently the negative expected second derivative of the log-likelihood. It measures how much information the data carry about θ: high information means the likelihood is sharply peaked and the MLE will be precise. The Cramér–Rao inequality states that no unbiased estimator can have variance smaller than 1/I(θ), and the MLE achieves this bound asymptotically. In practice, the inverse of the observed information matrix provides the asymptotic covariance matrix of the MLE, and its square roots are the reported standard errors.

MAP estimation is the Bayesian analogue of MLE. While MLE maximises the likelihood L(θ | x), MAP maximises the posterior p(θ | x) ∝ L(θ | x) · p(θ), where p(θ) is a prior distribution over the parameters. When the prior is flat (uniform), MAP and MLE coincide. MAP can be interpreted as a regularised form of MLE: for example, a Gaussian prior on regression coefficients corresponds to ridge (L2) regularisation.

Bayesian estimation is preferable when you have genuine prior knowledge that should constrain inference, when sample sizes are small enough that the prior has meaningful influence, or when you need full posterior distributions rather than point estimates and asymptotic intervals. MLE is typically preferred when priors are unavailable or hard to justify, when computational simplicity matters, or when asymptotic sample sizes are met — because in large samples Bayesian posteriors concentrate at the MLE anyway.