Bayesian Ridge Regression
Bayesian Ridge Regression (MacKay Probabilistic Regularisation) · Also known as: BRR, Bayesian linear regression with automatic relevance determination, evidence approximation ridge, marginal likelihood ridge
Bayesian Ridge Regression is a probabilistic formulation of ridge regression, introduced by David J. C. MacKay in 1992, in which the regularisation strength and noise precision are not fixed by the analyst but are instead estimated automatically by maximising the marginal likelihood (evidence) of the observed data. The result is a full posterior distribution over the regression weights together with calibrated predictive uncertainty.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Bayesian Ridge Regression is appropriate when the analyst wants a regularised linear model but wishes to avoid manual tuning of the penalty via cross-validation, or when calibrated predictive uncertainty intervals are required alongside point predictions. It assumes a linear relationship between features and response, Gaussian noise, and that a spherical Gaussian prior on the weights is a reasonable shrinkage assumption. It is suitable for low-to-moderate dimensional settings (up to a few thousand features with at least 20 observations) and is preferable to ordinary ridge regression whenever the noise level is uncertain or the regularisation strength is expected to vary across datasets. It is not appropriate when sparsity is required (use ARD regression or Lasso), when the relationship is strongly non-linear, or when n is very large and computational tractability of the evidence optimisation matters.
Strengths & limitations
- Automatic regularisation: alpha and beta are estimated from data, eliminating the need for a separate cross-validation grid search.
- Calibrated uncertainty: the posterior predictive distribution provides honest confidence intervals that account for both noise and weight uncertainty.
- Analytical tractability: the Gaussian-Gaussian conjugacy yields closed-form posterior and evidence expressions — no MCMC or variational approximations required.
- Robust to weak signals: the evidence procedure avoids over-fitting even when predictors outnumber useful signals, because over-complex models are penalised by the marginal likelihood.
- Interpretable coefficients: posterior means serve as regularised point estimates, while posterior variances quantify how well each coefficient is determined by the data.
- Linearity assumption: the method inherits all limitations of linear models; it cannot capture non-linear feature-response relationships without explicit feature engineering.
- Spherical prior: the isotropic Gaussian prior applies uniform shrinkage to all weights, which is suboptimal when feature relevances differ widely (ARD regression addresses this).
- Convergence not guaranteed: the evidence re-estimation is an EM-like procedure that converges to a local maximum of the marginal likelihood; different initialisations may give different hyperparameter solutions.
- Computational cost: forming and inverting S_N costs O(p^3) in the number of features p, making it impractical for very high-dimensional data without additional approximations.
- Gaussian noise assumption: heavy-tailed noise or outliers can distort the evidence-optimised hyperparameters; robust Bayesian alternatives or outlier pre-processing should be considered.
Frequently asked
How does Bayesian Ridge Regression differ from ordinary ridge regression?
Ordinary ridge regression requires the analyst to choose the penalty parameter lambda (equivalently alpha/beta) via cross-validation. Bayesian Ridge Regression treats both alpha and beta as unknowns and estimates them by maximising the marginal likelihood of the observed data. The posterior mean of the weights is equivalent to the ridge estimate at the optimised alpha/beta ratio, but the Bayesian formulation additionally delivers a posterior distribution over the weights and calibrated predictive intervals.
Do I need to scale my features before fitting?
Yes. The isotropic Gaussian prior assigns equal prior variance to every weight regardless of the scale of the corresponding feature. If features are on very different scales, the prior implicitly applies different amounts of shrinkage in the original feature space. Standardising features to zero mean and unit variance before fitting ensures the prior is truly non-informative about feature relevance and makes the hyperparameter estimation more stable.
When should I use ARD regression instead?
Automatic Relevance Determination (ARD) regression extends Bayesian Ridge Regression by assigning each feature its own precision parameter alpha_j. This allows the model to shrink irrelevant features to near zero while leaving relevant ones largely unpunished. Use ARD when you expect that only a subset of features is truly predictive and a sparse solution is desirable. ARD is more computationally demanding and can be sensitive to local optima, so Bayesian Ridge Regression is a safer default when sparsity is not a primary goal.
How do I interpret the predictive uncertainty?
For a new observation x_*, the predictive variance has two components. The term beta^{-1} represents aleatoric uncertainty — irreducible noise that would remain even with infinite training data. The term x_*' * S_N * x_* represents epistemic uncertainty — uncertainty about the true weights given the finite training set. Points far from the training distribution have large epistemic uncertainty, so the model signals that it is extrapolating. Both components are automatically calibrated through the evidence procedure.
Sources
- MacKay, D. J. C. (1992). Bayesian Interpolation. Neural Computation, 4(3), 415–447. DOI: 10.1162/neco.1992.4.3.415 ↗
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning (Ch. 3). Springer. ISBN: 978-0-387-31073-2
How to cite this page
ScholarGate. (2026, June 3). Bayesian Ridge Regression (MacKay Probabilistic Regularisation). ScholarGate. https://scholargate.app/en/machine-learning/bayesian-ridge-regression
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Elastic NetMachine learning↔ compare
- Lasso RegressionMachine learning↔ compare
- Ridge RegressionMachine learning↔ compare