Online Support Vector Machine
Also known as: Online SVM, Incremental SVM, LASVM, Pegasos SVM
Online SVM adapts the classical support vector machine to streaming or sequentially arriving data by updating the decision boundary one example at a time rather than solving a global quadratic program. Algorithms such as Pegasos and LASVM make this tractable at large scale, preserving the margin-maximising spirit of SVMs with sub-linear time per update.
Key highlights
- Processes one example per update in O(d) time for linear kernels, scaling to billions of examples.
- Adapts naturally to concept drift by weighting recent observations more heavily through its learning rate schedule.
- Inherits the strong theoretical generalization guarantees of the maximum-margin framework.
- Memory footprint can be bounded by budget SVM techniques that cap the support vector set size.
- Pegasos and similar solvers converge to within epsilon of the batch optimum in O(1/epsilon) iterations.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Choose Online SVM when data arrive in a stream or are too large to fit in memory at once, when the concept may drift slowly over time and the model should adapt without full retraining, or when strict latency requirements make batch SVM infeasible. It is well-suited to binary text classification, click-through prediction, and sensor-stream anomaly detection. Avoid it when the kernel must be exact and the support vector set grows unboundedly (use approximations or budget methods), when classes are severely imbalanced without reweighting, or when you need a well-calibrated probability estimate rather than a margin score — in those cases logistic regression or gradient-boosted trees are more convenient.
Strengths & limitations
- Processes one example per update in O(d) time for linear kernels, scaling to billions of examples.
- Adapts naturally to concept drift by weighting recent observations more heavily through its learning rate schedule.
- Inherits the strong theoretical generalization guarantees of the maximum-margin framework.
- Memory footprint can be bounded by budget SVM techniques that cap the support vector set size.
- Pegasos and similar solvers converge to within epsilon of the batch optimum in O(1/epsilon) iterations.
- Exact kernel SVMs require storing all support vectors, causing memory growth; budget approximations sacrifice some accuracy.
- Hyperparameter tuning (lambda, kernel bandwidth) requires held-out validation and cannot be done by cross-validation on a stream without extra bookkeeping.
- Does not naturally output calibrated probabilities; Platt scaling must be applied separately.
- Multiclass extension requires one-vs-one or one-vs-rest decompositions, multiplying cost.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
Is Online SVM the same as stochastic gradient descent on hinge loss?
Functionally yes for the linear case: Pegasos is a stochastic sub-gradient method on the regularised hinge loss. The difference lies in the regularisation schedule and the convergence proof; the decision boundary is the same max-margin hyperplane the batch SVM finds.
How do I choose the regularisation parameter lambda?
Lambda is inversely related to the classical C parameter (lambda = 1/(n*C)). Use a held-out stream window or periodic validation batches to tune it. Libraries like scikit-learn's SGDClassifier expose this as 'alpha'.
Can Online SVM handle non-linear boundaries?
Yes, via kernel tricks (Gaussian, polynomial), but the support vector set may grow without bound. Budget SVM methods or the Nyström random feature approximation keep cost manageable at the price of approximate kernels.
How does Online SVM compare to online logistic regression?
Online SVM maximises a geometric margin and handles non-separable data through hinge loss, while online logistic regression maximises log-likelihood and directly outputs probabilities. For calibrated scores, prefer logistic; for pure classification accuracy on linearly separable problems, SVM often edges it out.
Does concept drift break the model?
Mild drift is absorbed naturally as the decaying learning rate gives recent examples proportionally more influence. For severe or abrupt drift, couple Online SVM with a change-detection trigger that resets the weight vector.
Sources
- 1.Shalev-Shwartz, S., Singer, Y., Srebro, N., & Cotter, A. (2011). Pegasos: Primal estimated sub-gradient solver for SVM. Mathematical Programming, 127(1), 3–30.
- 2.Bordes, A., Ertekin, S., Weston, J., & Bottou, L. (2005). Fast kernel classifiers with online and active learning. Journal of Machine Learning Research, 6, 1579–1619.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 3). Online Support Vector Machine. ScholarGate. https://scholargate.app/machine-learning/online-support-vector-machine