Self-supervised Naive Bayes
Self-supervised Naive Bayes (EM-augmented Generative Classifier) · Also known as: Self-training Naive Bayes, EM Naive Bayes, Expectation-Maximization Naive Bayes, Pseudo-label Naive Bayes
Self-supervised Naive Bayes extends the classic Naive Bayes classifier to exploit large pools of unlabeled data by iteratively assigning soft pseudo-labels through an Expectation-Maximization loop. Originally demonstrated for text classification by Nigam et al. (2000), the approach can substantially improve accuracy when labeled examples are scarce but unlabeled data are plentiful.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Choose self-supervised Naive Bayes when labeled data are limited (dozens to hundreds of examples) but unlabeled data are abundant — a common scenario in text classification, spam filtering, document categorization, and clinical NLP where annotation is expensive. It is particularly effective when data follow a multinomial distribution (word counts, event frequencies) and when the Naive Bayes independence assumption is a reasonable approximation. Avoid it when labeled data are already sufficient (hundreds of well-balanced examples per class), when features are strongly correlated (violating the independence assumption), when the unlabeled data distribution is substantially different from the labeled set, or when interpretability demands a fully supervised, auditable model.
Strengths & limitations
- Efficiently leverages abundant unlabeled data, often improving accuracy significantly when labeled examples are scarce.
- Computationally inexpensive: the EM M-step for Naive Bayes is a single closed-form pass, scaling to very large unlabeled corpora.
- Probabilistic outputs provide calibrated class probabilities, not just hard labels.
- The unlabeled-data weight hyperparameter allows fine-grained control over how much the unsupervised signal influences the model.
- Well-suited to high-dimensional discrete feature spaces such as text bag-of-words representations.
- Simple to implement on top of any standard Naive Bayes library.
- Performance degrades if the independence assumption is heavily violated, a known weakness of the Naive Bayes family.
- If the initial supervised model is poor, the EM loop can converge to a bad local optimum that amplifies early errors.
- The approach assumes unlabeled data are drawn from the same distribution as labeled data; covariate shift can hurt performance.
- Does not natively handle continuous features without discretization or Gaussian Naive Bayes adaptations.
- The unlabeled-data weight must be tuned; setting it too high causes the model to be dominated by noise in the unlabeled pool.
Frequently asked
How many labeled examples are needed to start?
The approach was shown to work with as few as 10–20 labeled examples per class in text settings, though 50 or more per class generally yields a better initialization and more stable EM convergence.
What is the unlabeled-data weight and how should I set it?
The weight (often denoted lambda) scales the contribution of unlabeled examples relative to labeled ones. Nigam et al. recommend values between 0.1 and 1.0, tuned via cross-validation on the labeled set; a weight of 1.0 treats labeled and unlabeled data equally.
How is this different from semi-supervised Naive Bayes?
The distinction is largely terminological. Self-supervised Naive Bayes emphasizes that the supervision signal (pseudo-labels) is generated by the model itself without human annotation, while semi-supervised Naive Bayes is the broader category. In practice both refer to EM-augmented Naive Bayes with unlabeled data.
Does EM always improve performance over supervised Naive Bayes?
Not always. If the independence assumption is badly violated, the unlabeled pool introduces noise, or the data distributions differ, EM can hurt. Evaluate with a held-out labeled test set before deploying.
Can this be applied to continuous features?
Yes, by using Gaussian Naive Bayes as the base model. The EM structure is identical; only the likelihood function changes from multinomial to Gaussian. Discretization of continuous features is an alternative that keeps the multinomial formulation.
Sources
- Nigam, K., McCallum, A. K., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39(2-3), 103–134. DOI: 10.1023/A:1007692713085 ↗
- Chapelle, O., Scholkopf, B., & Zien, A. (Eds.) (2006). Semi-supervised Learning. MIT Press. ISBN: 978-0-262-03358-9
How to cite this page
ScholarGate. (2026, June 3). Self-supervised Naive Bayes (EM-augmented Generative Classifier). ScholarGate. https://scholargate.app/en/machine-learning/self-supervised-naive-bayes
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Naive BayesMachine learning↔ compare
- Self-supervised LearningMachine learning↔ compare
- Self-supervised Logistic RegressionMachine learning↔ compare
- Semi-supervised LearningMachine learning↔ compare
- Semi-supervised Naive BayesMachine learning↔ compare