Stepwise Regression

Forward selection begins with an empty model and adds the predictor with the highest F-to-enter at each step, stopping when no remaining variable clears the entry threshold. Backward elimination begins with all p predictors and removes the predictor with the lowest F-to-remove at each step, stopping when all retained variables exceed the removal threshold. Bidirectional stepwise selection combines both passes: after each addition it checks whether any previously included variable should be removed, and vice versa. Bidirectional selection is generally preferred over pure forward selection because it can correct early inclusions, but all three variants share the same post-selection inference problems.

Standard p-values assume the model structure was fixed before seeing the data. In stepwise regression the data are used to choose which predictors appear in the model and then again to estimate and test those predictors. This double-use inflates the apparent significance of retained variables — an effect sometimes called the 'winner's curse'. Simulation studies show that variables with true zero effect are regularly selected and appear significant when stepwise is applied to moderate samples. Correcting for this requires either data splitting, bootstrap-based validation, or penalised estimation methods that fold selection and estimation into a single regularised optimisation.

Stepwise selection is most defensible in purely predictive applications where (a) the dataset is large relative to the number of candidates, (b) the resulting model will be validated on a held-out or prospective sample, and (c) inference about individual coefficients is not the goal. In exploratory work it can serve as a preliminary screening tool, provided the shortlisted variables are then evaluated in a pre-specified confirmatory model on new data. It should not be used for causal inference, hypothesis testing about specific predictors, or any setting where the reported p-values will be interpreted at face value.

For sparse variable selection with valid inference, LASSO (Tibshirani, 1996) and elastic net (Zou & Hastie, 2005) are the standard alternatives: they fold selection and estimation into a single penalised criterion, their tuning parameter is chosen by cross-validation, and bootstrap confidence intervals can be reported. Best-subset selection (examined systematically with modern algorithms) provides an exhaustive alternative when p is not too large. For confirmatory research, the correct approach is to specify the model on theoretical grounds before data collection, not to let the data choose the predictors.