Regularized Random Forest
Regularized Random Forest (RRF) · Also known as: RRF, Guided Regularized Random Forest, GRRF, regularized tree ensemble
Regularized Random Forest (RRF), introduced by Deng and Runger in 2012, extends the standard Random Forest by adding a penalty that discourages splits on features not already used in the ensemble. This built-in regularization produces sparser, less redundant feature subsets, making the model especially valuable when feature selection is as important as predictive accuracy.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use Regularized Random Forest when you need both high predictive accuracy and a compact, interpretable feature subset — for example in high-dimensional biomarker discovery, genomics, or any domain where the number of candidate features greatly exceeds the sample size. It is especially well-suited when standard Random Forest produces unstable or bloated importance rankings. Do not use RRF when all features are known to be informative and no selection is desired, when the dataset is very small (below ~50 observations) and cross-validated performance is needed, or when full model transparency with explicit coefficients is required (use regularized logistic or linear regression instead). Tune lambda carefully: too small a value aggressively excludes features that are genuinely useful.
Strengths & limitations
- Performs feature selection and prediction in a single pass, eliminating the need for a separate variable-selection step.
- Produces sparser, more stable feature-importance rankings than standard Random Forest, especially on high-dimensional data.
- Inherits the robustness and accuracy of Random Forest — resistant to overfitting through bagging and tree diversity.
- Handles mixed feature types (continuous, categorical, binary) and nonlinear interactions without distributional assumptions.
- The guided variant (GRRF) allows domain knowledge to be injected as a prior importance score, steering selection toward theoretically meaningful features.
- Adds one or two hyperparameters (lambda and, in GRRF, the importance weight) that require careful tuning, typically via cross-validation.
- Computationally more expensive than standard Random Forest because the globally shared feature-use memory must be updated across trees.
- On small datasets (n below ~50) both the regularization and the out-of-bag error estimate become unreliable.
- Still a black-box ensemble; does not provide explicit coefficients or confidence intervals for individual feature effects.
- Very small lambda values can over-select a single dominant feature, producing a degenerate forest with high variance.
Frequently asked
What does the lambda parameter control?
Lambda (between 0 and 1) penalizes features that have not yet appeared in the forest. A lambda of 1 gives standard Random Forest behavior. Smaller values impose stronger sparsity, pushing the forest to reuse already-selected features. A value around 0.5–0.8 is a common starting range, tuned by cross-validation.
How is RRF different from standard Random Forest with post-hoc feature selection?
Standard Random Forest selects features after fitting by ranking importance scores, then retrains a model on the top features — a two-stage process that can be unstable. RRF integrates selection into the splitting criterion, so the feature set and the model are optimized jointly, typically yielding more stable and parsimonious results.
When should I use the Guided variant (GRRF) instead of plain RRF?
Use GRRF when you have prior knowledge — for example, published literature or domain expertise — indicating which features are likely informative. GRRF lets you supply prior importance scores that bias the regularization, so theoretically motivated features are favored during selection.
Does RRF work for regression as well as classification?
Yes. The regularized gain criterion applies to any impurity measure, including variance reduction for regression tasks. The aggregation step uses the mean of tree predictions rather than a majority vote.
Is the out-of-bag error still valid in RRF?
OOB error is available but can be optimistically biased when lambda is very small, because aggressive regularization may overfit the feature selection to the training trees. Cross-validation on a held-out test set is recommended alongside OOB error to confirm generalization.
Sources
- Deng, H., & Runger, G. (2012). Feature selection via regularized trees. Proceedings of the 2012 International Joint Conference on Neural Networks (IJCNN), IEEE, pp. 1–8. DOI: 10.1109/IJCNN.2012.6252640 ↗
- Deng, H., & Runger, G. (2013). Gene selection with guided regularized random forest. Pattern Recognition, 46(12), 3483–3489. DOI: 10.1016/j.patcog.2013.05.018 ↗
How to cite this page
ScholarGate. (2026, June 3). Regularized Random Forest (RRF). ScholarGate. https://scholargate.app/en/machine-learning/regularized-random-forest
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Decision TreeMachine learning↔ compare
- Extra TreesMachine learning↔ compare
- Random ForestMachine learning↔ compare
- Regularized Decision TreeMachine learning↔ compare
- Regularized Gradient BoostingMachine learning↔ compare