Gradient Boosting
Gradient Boosting Machine (Friedman's Gradient Boosting) · Also known as: Gradient Boosting (GBM), GBM, gradient boosted trees, gradient boosting machine
Gradient Boosting is an ensemble learning method, formalised by Jerome H. Friedman in 2001, that combines a sequence of weak learners — typically shallow decision trees — so that each new tree is fitted to minimise the residual errors of the trees before it. It is the core algorithm behind popular implementations such as XGBoost, LightGBM and CatBoost.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+38 more
When to use it
A strong choice for accurate classification and prediction on cross-sectional or panel tabular data with at least about 100 observations, across continuous, categorical, binary and ordinal features. Feature scaling is generally not required, but hyperparameter tuning of the number of trees, learning rate and maximum depth is critical, and early stopping is recommended to guard against overfitting. With fewer than about 100 observations the sequential trees tend to memorise noise, and Random Forest or a single decision tree is safer.
Strengths & limitations
- Very high predictive accuracy on tabular data for both classification and prediction.
- Feature scaling is generally unnecessary, and it handles continuous, categorical, binary and ordinal features.
- Forms the core algorithm behind widely used libraries such as XGBoost, LightGBM and CatBoost.
- Early stopping and a small learning rate give fine control over the bias–variance trade-off.
- Hyperparameter tuning of the number of trees, learning rate and maximum depth is critical and time-consuming.
- Sequential boosting is prone to overfitting if not regularised, which is why early stopping is recommended.
- On small samples (n below about 100) the sequential trees memorise noise, and even early stopping is insufficient.
- With too little data there is not enough cross-validation material to optimise hyperparameters reliably.
Frequently asked
How is Gradient Boosting different from Random Forest?
Random Forest builds many independent trees in parallel and averages them, whereas Gradient Boosting builds trees sequentially, each one fitted to correct the residual errors of the ones before it. Boosting often reaches higher accuracy but is more sensitive to overfitting and needs careful tuning.
Why is early stopping recommended?
Because each new tree chases the remaining error, an over-long boosting run starts fitting noise. Early stopping halts training once validation performance stops improving, which is the main safeguard against overfitting.
Do I need to scale my features?
Feature scaling is generally not required for tree-based gradient boosting, since splits depend on order rather than magnitude. The critical knobs are instead the number of trees, the learning rate and the tree depth.
What sample size do I need?
At least about 100 observations. Below that, the sequential trees memorise noise and even early stopping is insufficient, so Random Forest or a single decision tree is the safer choice.
Sources
- Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics, 29(5), 1189–1232. DOI: 10.1214/aos/1013203451 ↗
How to cite this page
ScholarGate. (2026, June 1). Gradient Boosting Machine (Friedman's Gradient Boosting). ScholarGate. https://scholargate.app/en/machine-learning/gradient-boosting
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Decision TreeMachine learning↔ compare
- LightGBMMachine learning↔ compare
- Logistic RegressionResearch Statistics↔ compare
- Random ForestMachine learning↔ compare
- XGBoostMachine learning↔ compare