TF-IDF — Term Frequency–Inverse Document Frequency
Term Frequency–Inverse Document Frequency Vectorization · Also known as: term weighting, tf-idf weighting, TF-IDF Vektörizasyonu
TF-IDF, introduced by Salton and Buckley (1988), is a term-weighting scheme that scores each word in a document by how often it appears there and how rare it is across the whole collection. It turns raw text into weighted document vectors, giving high weight to terms that are frequent in one document but uncommon elsewhere.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+13 more
When to use it
Use TF-IDF when you have a text corpus and need to turn documents into numeric features for retrieval, classification, or similarity comparison. It relies on the bag-of-words assumption and requires text preprocessing beforehand. The collection should be reasonably large — with under about 100 documents the IDF weights become unstable and a simple word-frequency analysis is more reliable. With no text data at all, TF-IDF cannot be computed.
Strengths & limitations
- Simple, fast, and interpretable — each weight is directly traceable to local frequency and global rarity.
- Turns raw text into numeric document vectors usable by search, classification, and clustering methods.
- Downweights ubiquitous, uninformative words while highlighting terms that distinguish a document.
- Rests on the bag-of-words assumption and ignores word order and context.
- IDF weights become unstable and meaningless on very small corpora.
- Requires careful text preprocessing; noisy tokens degrade the weights.
Frequently asked
What do TF and IDF each measure?
Term frequency (TF) measures how often a term appears within a single document, capturing how central it is locally. Inverse document frequency (IDF) measures how rare the term is across the whole collection, downweighting words that appear in many documents. Multiplying them gives high weight to terms that are frequent in one document but uncommon elsewhere.
How large does my corpus need to be?
TF-IDF needs a reasonably large collection — roughly 100 documents or more. Below that, IDF weights become unstable and lose meaning, and a simpler word-frequency analysis is the more reliable choice.
Does TF-IDF understand meaning or context?
No. TF-IDF rests on the bag-of-words assumption: it weights individual terms by frequency and rarity and ignores word order, syntax, and semantics. For context or semantic similarity, embedding methods such as Word2Vec are more appropriate.
What preprocessing does it require?
The corpus must be cleaned and tokenised consistently before weighting — normalising text and splitting it into comparable term units. Skipping this step leaves noisy tokens that degrade the weights.
Sources
- Salton, G. & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513-523. DOI: 10.1016/0306-4573(88)90021-0 ↗
How to cite this page
ScholarGate. (2026, June 1). Term Frequency–Inverse Document Frequency Vectorization. ScholarGate. https://scholargate.app/en/text-mining/tf-idf
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Sentiment AnalysisText mining↔ compare
- Text ClassificationText mining↔ compare
- Word2VecText mining↔ compare