Supervised Text Classification

There is no fixed number — it depends on the number and balance of categories, the difficulty of the distinctions, and the richness of the features. Simple binary tasks with clear cues can work with a few hundred labeled documents, while many fine-grained or imbalanced categories may need thousands. The practical test is the learning curve: label in batches, evaluate out-of-sample performance as the training set grows, and stop when F1 plateaus. Reliable labels matter more than sheer volume, since the model cannot exceed the quality of its training codes.

Because an imperfect classifier produces biased proportions when you simply count its predictions: if it systematically over- or under-predicts a category, the corpus-level estimate inherits that bias even when accuracy looks decent. Hopkins and King's method targets the quantity most social-science questions actually want — the share of documents in each category — and corrects for the classifier's measured error rates, yielding approximately unbiased proportions without needing every individual document classified correctly. This is the right approach when your inference is about aggregates rather than specific documents.

A dictionary method applies a fixed, analyst-specified word list and needs no training data, trading flexibility for transparency and portability. Supervised classification learns the text-to-category mapping from hand-labeled examples, so it captures context and non-obvious cues and usually performs better within its training domain, at the cost of requiring labeled data and risking poor generalization. Dictionaries are auditable but brittle across domains; classifiers are accurate in-domain but data-hungry. Many projects use both — a dictionary baseline plus a supervised model — and report validated performance for each. See dictionary-based-text-analysis.