The Bias–Variance Tradeoff

The Concept and Its Formulation

The expected prediction error decomposes formally as: Expected Error = Bias² + Variance + Irreducible Noise. Bias measures how far the average prediction of a model is from the true value; fitting a linear model to a nonlinear relationship is a canonical illustration. Variance captures how much model predictions fluctuate across different training sets drawn from the same population. The third term, irreducible noise, represents randomness inherent in the data that no model can remove regardless of complexity. Understanding this decomposition is essential because only the first two components are within a researcher's control.

Relationship to Complexity and How to Assess It

As model complexity increases, bias falls but variance rises; this inverse relationship is the heart of the tradeoff. In practice, the balance is assessed through cross-validation: data is split into multiple folds, the model is evaluated on each held-out portion, and average test error is computed across folds. The complexity level yielding the lowest test error identifies the optimal model. A large gap between training error and test error signals high variance, whereas both errors remaining high together signals high bias. Regularisation methods such as ridge regression or dropout penalise complexity and shift the tradeoff toward lower variance.

Common Misunderstandings

The most common error is assuming that greater complexity always yields better performance. A model that fits training data perfectly may perform poorly on new data; this is overfitting, the direct consequence of high variance. A second frequent mistake is treating bias and variance as independently controllable: they are coupled, and reducing one tends to increase the other. Finally, large datasets do not guarantee low variance on their own; more data helps reduce variance but the functional form chosen for the model remains the primary determinant of bias. Researchers should resist equating model complexity with model quality.

Why It Matters and How to Report It

The bias–variance tradeoff is a foundational principle of statistical modelling, not merely a machine learning concept. When reporting results, researchers should present performance on an independent test set or via cross-validation rather than training performance alone. The complexity level of the chosen model, such as a regularisation parameter or tree depth, should be stated along with the rationale for the selection. Training and test errors should be reported side by side; a large discrepancy warrants an explicit discussion of overfitting risk. This transparency allows readers to judge how well findings are likely to generalise beyond the sample.

Sources