Regularized Support Vector Machine
Regularized Support Vector Machine (L1/L2-penalized SVM) · Also known as: Regularized SVM, L1-SVM, L2-SVM, penalized SVM
Regularized Support Vector Machine extends the classic SVM by explicitly controlling the trade-off between margin maximization and training error through an L1 or L2 penalty parameter. The soft-margin formulation introduced by Cortes and Vapnik in 1995 is itself a regularized model, and later L1-SVM variants additionally promote feature sparsity, enabling automatic variable selection in high-dimensional settings.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use Regularized SVM when you need a high-accuracy binary or multi-class classifier on tabular or text data with moderate dimensionality, or when you want automatic feature selection via L1-SVM in high-dimensional settings such as genomics or text classification. SVMs are particularly effective when the number of features is large relative to observations. Do not use when full probabilistic output (calibrated probabilities) is required without an additional calibration step, when data are very large (n > 100 000) since training cost scales poorly, or when the problem demands easy interpretability of individual feature coefficients by non-technical stakeholders.
Strengths & limitations
- Effective in high-dimensional spaces where the number of features exceeds or approaches the number of observations.
- L1-SVM provides built-in sparse feature selection, reducing dimensionality automatically.
- The kernel trick enables nonlinear classification without explicitly computing high-dimensional feature maps.
- Convex optimization guarantees a globally optimal solution with no local minima.
- Robust to outliers relative to logistic regression because only support vectors define the boundary.
- Training complexity scales roughly quadratically to cubically with sample size, making large datasets slow without specialized solvers.
- Does not natively output calibrated class probabilities — Platt scaling or isotonic regression is required.
- Requires careful tuning of C (and kernel parameters) via cross-validation; defaults are rarely optimal.
- Interpretation is limited compared to logistic regression — the support-vector formulation does not yield simple odds ratios or marginal effects.
Frequently asked
What does the C parameter actually control?
C is the inverse of the regularization strength. A large C tells the SVM to penalize misclassifications heavily, producing a narrow margin that fits the training data more closely. A small C allows more margin violations and produces a wider, more general boundary. Tuning C via cross-validation is essential.
When should I prefer L1-SVM over the standard (L2) SVM?
Choose L1-SVM when you need a sparse solution — that is, when you want the model to automatically set many feature weights to zero. This is valuable in high-dimensional settings such as text mining or genomics where most features are irrelevant. L2-SVM typically achieves better accuracy when most features are informative.
Can Regularized SVM output probabilities?
Not natively. The decision function gives a signed margin score, not a calibrated probability. Platt scaling — fitting a sigmoid to the margin scores on a held-out set — is the standard post-processing step to obtain class probabilities.
Does regularization prevent overfitting in SVMs?
Yes. Regularization through C controls model complexity. A properly tuned C prevents the model from fitting noise in the training data. On high-dimensional data, L1 regularization additionally reduces the effective model complexity by zeroing out irrelevant features.
What kernel should I start with?
Start with a linear kernel — it is fast, interpretable, and often competitive in high-dimensional or text settings. If the linear kernel falls short on validation, try the RBF kernel and tune both C and gamma. Polynomial kernels are less common and harder to tune.
Sources
- Cortes, C. & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. DOI: 10.1007/BF00994018 ↗
- Zhu, J., Rosset, S., Tibshirani, R. & Hastie, T. (2004). 1-norm support vector machines. Advances in Neural Information Processing Systems (NIPS), 16. link ↗
How to cite this page
ScholarGate. (2026, June 3). Regularized Support Vector Machine (L1/L2-penalized SVM). ScholarGate. https://scholargate.app/en/machine-learning/regularized-support-vector-machine
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Lasso RegressionMachine learning↔ compare
- Linear Discriminant AnalysisMachine learning↔ compare
- Regularized linear regressionMachine learning↔ compare
- Regularized Logistic RegressionMachine learning↔ compare