Latent Semantic and Topic Models

Latent semantic and topic models represent documents by hidden themes rather than surface words, capturing semantic relationships and easing the vocabulary mismatch between queries and documents.

Definition

Latent semantic and topic models are dimensionality-reduction and generative methods that represent documents as combinations of a small number of latent dimensions or topics, derived from co-occurrence structure in the term-document matrix, so that semantically related terms and documents lie close together.

Scope

This topic covers methods that uncover latent structure in text: latent semantic analysis (also called latent semantic indexing) via truncated singular value decomposition of the term-document matrix, probabilistic latent semantic indexing, and latent Dirichlet allocation and related probabilistic topic models. It addresses how these projections capture synonymy and semantic similarity, how topics are interpreted, and how the representations support retrieval and browsing. It excludes general matrix-factorization and neural-embedding methods beyond their use as semantic text representations.

Core questions

• How does truncated singular value decomposition produce a latent semantic space?
• How do latent representations address synonymy and vocabulary mismatch?
• How do probabilistic topic models such as LDA generate documents from topics?
• How are the resulting topics interpreted and labeled?
• How do latent representations improve retrieval, browsing, and similarity?

Key concepts

• latent semantic analysis / indexing
• term-document matrix
• truncated singular value decomposition
• dimensionality reduction
• synonymy and polysemy
• probabilistic latent semantic indexing
• latent Dirichlet allocation
• topic-word and document-topic distributions

Key theories

Latent semantic analysis

Applying a truncated singular value decomposition to the term-document matrix projects documents and terms into a low-dimensional latent space where semantically related items are close, mitigating synonymy and capturing higher-order co-occurrence.

Probabilistic topic models

Probabilistic latent semantic indexing and latent Dirichlet allocation model each document as a mixture of latent topics, each a distribution over words, providing a generative, interpretable account of document content.

Clinical relevance

Latent and topic models support semantic search, document similarity, recommendation, and corpus exploration by theme, helping match concepts rather than exact words. They are conceptual predecessors of dense neural embeddings, which now provide learned semantic representations for retrieval at scale.

History

Latent semantic analysis was introduced in 1990 to overcome vocabulary mismatch via matrix decomposition. Hofmann's 1999 probabilistic latent semantic indexing gave a generative reformulation, and Blei, Ng, and Jordan's 2003 latent Dirichlet allocation established Bayesian topic modeling, which became a major tool for analyzing large text corpora.

Key figures

• Susan Dumais
• Thomas Landauer
• Thomas Hofmann
• David Blei

Seminal works

• deerwester1990
• hofmann1999
• blei2003

Frequently asked questions

How do latent semantic models help with vocabulary mismatch?

By projecting documents and terms into a shared latent space based on co-occurrence, these models place synonyms and related terms close together. A query and a relevant document can then match through shared latent dimensions even if they use different words for the same concept.

What does latent Dirichlet allocation actually produce?

LDA learns a set of topics, each a distribution over words, and represents every document as a mixture of those topics. This gives interpretable themes and a compact document representation useful for organizing, searching, and analyzing large collections.