Kullback-Leibler Divergence
Kullback-Leibler Information Divergence · Also known as: KL divergence, relative entropy, information divergence
Kullback-Leibler divergence, also called relative entropy or information divergence, measures the asymmetric difference between two probability distributions. Introduced by Solomon Kullback and Richard Leibler in 1951, this information-theoretic measure quantifies how one probability distribution diverges from a reference distribution, ranging from 0 (identical distributions) to infinity. It is foundational in information theory, machine learning, and decision-making with probabilistic frameworks.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
KL divergence is appropriate when comparing probability distributions, fitting models to observed data, or assessing information loss in approximations. Use it in machine learning (variational inference, generative models), information theory, decision-making under uncertainty, and statistical hypothesis testing. Requires positive, normalized probability values. Not suitable for comparing raw counts without normalization or for symmetric distance applications.
Strengths & limitations
- Theoretically grounded in information theory; directly measures information loss
- Sensitive to differences in distribution shape and tail behavior
- Widely used in machine learning, particularly in probabilistic inference and deep learning
- Natural interpretation in terms of bits or nats of information
- Asymmetric: KL(P||Q) ≠ KL(Q||P); choice of reference distribution matters
- Undefined when Q is zero and P is nonzero for any outcome (infinite divergence)
- Not a true metric; does not satisfy triangle inequality or symmetry
- Sensitive to small probabilities; can be dominated by rare events
Frequently asked
Why is KL divergence asymmetric?
KL divergence measures the cost of assuming Q when the truth is P. Reversing P and Q changes the cost function: KL(P||Q) penalizes missing mass (P is nonzero, Q is not), while KL(Q||P) penalizes false mass (Q is nonzero, P is not). These produce different results because they represent different information-theoretic scenarios.
How do I handle zero probabilities in KL divergence?
If Q has zeros where P is nonzero, KL divergence is undefined (infinite). Use add-one smoothing or pseudocounts to prevent exact zeros. Alternatively, use Jensen-Shannon divergence, which is symmetric and handles this case gracefully.
What is the difference between KL divergence and cross-entropy?
Cross-entropy measures the average bits needed to encode events from P using a code optimized for Q. KL divergence is the difference between cross-entropy and entropy: KL(P||Q) = H(P,Q) - H(P), where H is entropy. KL divergence isolates the extra cost due to Q.
When should I use KL divergence over Jensen-Shannon divergence?
Use KL divergence when you have a clear reference (true) distribution and want to measure directional information loss. Use Jensen-Shannon when you need symmetry, want to treat both distributions equally, or want a true metric satisfying triangle inequality.
Sources
- Kullback, S., & Leibler, R. A. (1951). On information and sufficiency. Annals of Mathematical Statistics, 22(1), 79-86. DOI: 10.1214/aoms/1177729694 ↗
- Cover, T. M., & Thomas, J. A. (1991). Elements of Information Theory. Wiley-Interscience. DOI: 10.1002/0471200611 ↗
How to cite this page
ScholarGate. (2026, June 3). Kullback-Leibler Information Divergence. ScholarGate. https://scholargate.app/en/decision-making/kullback-leibler-divergence
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Hellinger DistanceDecision-making↔ compare
- Jensen-Shannon DivergenceDecision-making↔ compare