Regularized Decision Tree
Regularized Decision Tree (Pruned and Constrained CART) · Also known as: pruned decision tree, cost-complexity pruned tree, penalized decision tree, constrained CART
A regularized decision tree is a decision tree model whose complexity is intentionally limited through pruning, depth constraints, or penalty terms to prevent overfitting. Rooted in Breiman et al.'s CART framework (1984), regularization converts the greedy tree-growing procedure into a bias-variance tradeoff, yielding models that generalize better to unseen data than fully-grown trees.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use a regularized decision tree when you need an interpretable, rule-based model on tabular data and want to avoid the overfitting that plagues fully-grown trees. It is well-suited to datasets with tens to low hundreds of features where stakeholders require transparent split logic. Prefer regularized decision trees over unregularized ones whenever sample sizes are modest, noise is present, or the model will be audited. Do not use when predictive accuracy is paramount and interpretability is secondary — ensemble methods such as random forests or gradient boosting will almost always outperform a single tree. Also avoid when there are fewer than about 30 observations, as cross-validated pruning becomes unreliable.
Strengths & limitations
- Produces an interpretable, human-readable set of if-then rules after pruning.
- Cost-complexity pruning is principled: it selects model complexity by cross-validation rather than ad hoc rules.
- Handles mixed feature types (continuous, categorical, binary) without preprocessing.
- Can model nonlinear relationships and feature interactions without explicit feature engineering.
- Regularization substantially reduces the variance of a single tree, improving generalization.
- Even well-pruned single trees are less accurate than ensemble methods such as random forests or gradient boosting.
- Trees are unstable: a small change in the data can lead to a very different tree structure.
- Optimal alpha search via cross-validation adds computational overhead over simple stopping rules.
- Greedy recursive splitting does not guarantee a globally optimal tree, even after pruning.
Frequently asked
What is cost-complexity pruning and how does alpha work?
Alpha (the complexity parameter) penalizes each additional leaf in the tree. Setting alpha=0 returns the full unpruned tree; larger alpha values remove more leaves. The best alpha is found by fitting the model for a range of values and picking the one with the lowest cross-validated error.
Is a regularized decision tree the same as a random forest?
No. A regularized decision tree is still a single tree — just constrained to prevent overfitting. A random forest grows many trees on bootstrap samples and averages their predictions, which achieves far lower variance at the cost of interpretability.
Which regularization approach should I start with?
Start with max_depth (try 3-5) and min_samples_leaf (try 5-20) as they are fast to tune. Then add cost-complexity pruning via cross-validated alpha search for a more principled result.
When does regularization not help a decision tree?
If the true decision boundary is genuinely complex and the dataset is large, regularization will simply underfit. In those cases, gradient boosting or random forests are better choices.
Can I regularize a tree for both classification and regression?
Yes. Cost-complexity pruning applies to both CART classification trees (using Gini or entropy impurity) and regression trees (using MSE), with the same cross-validation procedure.
Sources
- Breiman, L., Friedman, J., Olshen, R., & Stone, C. (1984). Classification and Regression Trees. Wadsworth. ISBN: 978-0-412-04841-8
- Esposito, F., Malerba, D., & Semeraro, G. (1997). A comparative analysis of methods for pruning decision trees. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(5), 476–491. DOI: 10.1109/34.589207 ↗
How to cite this page
ScholarGate. (2026, June 3). Regularized Decision Tree (Pruned and Constrained CART). ScholarGate. https://scholargate.app/en/machine-learning/regularized-decision-tree
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- BoostingMachine learning↔ compare
- Decision TreeMachine learning↔ compare
- Extra TreesMachine learning↔ compare
- Random ForestMachine learning↔ compare
- Regularized linear regressionMachine learning↔ compare
- Regularized random forestMachine learning↔ compare