CatBoost
Also known as: CatBoost (Categorical Boosting), categorical boosting, ordered boosting, kategorik gradyan artırma
CatBoost is a gradient boosting algorithm, introduced by Prokhorenkova and colleagues at Yandex in 2018, that handles categorical variables natively and uses ordered target encoding to avoid label leakage. By building an additive ensemble of trees while permuting the data order at each iteration, it is often superior to XGBoost and LightGBM on category-heavy data.
Key highlights
- Native handling of categorical features via cat_features — no one-hot encoding required.
- Ordered target encoding avoids label leakage that biases ordinary target encoding.
- Ordered boosting permutes the data each iteration, reducing overfitting.
- Symmetric trees give fast prediction and native GPU support competitive with LightGBM.
- Often outperforms XGBoost and LightGBM on category-heavy data.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
A strong fit for classification and prediction on tabular data dominated by categorical features, with at least about 100 observations. It does not require normally distributed data and accepts continuous, categorical, binary, and ordinal features, working on cross-sectional and panel-style structures. Hyperparameter tuning of depth (6-10), learning rate (0.01-0.1), and iterations (500-2000) is recommended. Below about 100 observations, simpler models such as logistic regression or a single decision tree are safer.
Strengths & limitations
- Native handling of categorical features via cat_features — no one-hot encoding required.
- Ordered target encoding avoids label leakage that biases ordinary target encoding.
- Ordered boosting permutes the data each iteration, reducing overfitting.
- Symmetric trees give fast prediction and native GPU support competitive with LightGBM.
- Often outperforms XGBoost and LightGBM on category-heavy data.
- On small samples (n below about 100) even ordered boosting overfits and early stopping becomes ineffective.
- Hyperparameter tuning (depth, learning rate, iterations) needs enough validation data, which small samples lack.
- As a boosted-tree ensemble it offers no explicit coefficients to interpret.
- Its advantage shrinks on data with few or no categorical features.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
Do I need to one-hot encode categorical variables?
No. CatBoost handles categorical variables natively when you mark them as cat_features, converting each category to a numeric target statistic on the fly, so manual one-hot encoding is unnecessary.
What is ordered boosting?
At each iteration CatBoost permutes the data order and computes a row's target encoding and gradient only from the observations preceding it. This avoids the label leakage of ordinary target encoding and reduces overfitting.
How does CatBoost compare to XGBoost and LightGBM?
CatBoost is generally superior on category-heavy data thanks to its native categorical handling, while its symmetric trees and GPU support stay competitive in speed. On data with few categorical features the advantage shrinks.
What sample size and tuning does it need?
Use at least about 100 observations; below that even ordered boosting overfits and early stopping is ineffective. Tune depth (6-10), learning rate (0.01-0.1), and iterations (500-2000) with a validation set.
Sources
- 1.Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V. & Gulin, A. (2018). CatBoost: Unbiased Boosting with Categorical Features. In NeurIPS 2018.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 1). CatBoost. ScholarGate. https://scholargate.app/machine-learning/catboost