Guides / Probability

Entropy, cross-entropy and KL divergence: weight by p, keep the minus sign, and match the log to the unit

Entropy measures how uncertain a distribution is, cross-entropy measures the cost of describing outcomes from one distribution with the probabilities of another, and the KL divergence is the gap between the two. Every one of them is an average of a log weighted by the true distribution p. Cross-entropy is the loss a classifier trains on, and the KL divergence is the extra term in a variational autoencoder. The mistakes come from swapping the two arguments, from mixing bits and nats, and from losing the minus sign in front of the sum.

Why it works

An outcome with probability p carries a surprise of −log p: a certain outcome surprises you not at all, and a one-in-a-million outcome a lot. Entropy is the average surprise when outcomes really come from p. If you use the wrong probabilities q to judge them, each outcome i still happens with probability p_i but costs −log q_i, so the average cost is the cross-entropy H(p, q). That is why the weights are always p: they are how often each outcome actually happens.

The KL divergence is the extra cost of using q instead of p, H(p, q) − H(p). Gibbs' inequality says it can't be negative, so no q describes data from p more cheaply than p itself. A classifier that minimizes cross-entropy is therefore pushing its q toward the labels' p, and because H(p) doesn't depend on the model, minimizing cross-entropy and minimizing KL are the same training problem.

Example 1: entropy in bits and nats, and perplexity

A source emits A, B, C with probabilities 1/2, 1/4, 1/4. Find its entropy in bits and in nats, and its perplexity.

1. −log₂(1/2) = 1, −log₂(1/4) = 2The surprise of each outcome, in bits.
2. H = (1/2)(1) + (1/4)(2)Weight each surprise by how often it happens …
3. + (1/4)(2) = 1.5 bits… and add the three terms.
4. 1.5 · ln 2 ≈ 1.040 natsMultiply by ln 2 to turn bits into nats; computing −Σ p ln p directly gives the same number.
5. 2^1.5 = e^1.040 ≈ 2.828The perplexity. The source is as unpredictable as a fair choice among about 2.8 outcomes: between 2 and 3, as it should be for three outcomes that aren't equally likely.

Example 2: the cross-entropy loss of a classifier

A three-example batch has one-hot labels. The model gives the true class probability 0.7, 0.5 and 0.9. Find the mean cross-entropy loss in nats.

1. H(y, q) = −Σ y_i log q_iThe cross-entropy of one example.
2. = −log q_trueA one-hot y is 1 on the true class and 0 elsewhere, so only one term survives. The probabilities on the wrong classes enter only through the softmax that made q.
3. −ln 0.7 ≈ 0.357One loss per example: the first …
4. −ln 0.5 ≈ 0.693… the second …
5. −ln 0.9 ≈ 0.105… and the third.
6. (0.357 + 0.693 + 0.105)/3Average the three losses.
7. mean loss ≈ 0.385 natsFrameworks use the natural log, so a loss curve is in nats. A single confident mistake, 0.1 on the true class, would cost −ln 0.1 ≈ 2.303 on its own.

Example 3: KL divergence as cross-entropy minus entropy

Let p = (0.5, 0.3, 0.2) and q = (0.25, 0.25, 0.5). Find H(p), H(p, q) and KL(p ‖ q).

1. H(p) = −(0.5 ln 0.5Entropy of the true distribution …
2. + 0.3 ln 0.3 + 0.2 ln 0.2)… one term per outcome …
3. ≈ 1.030… in nats.
4. H(p, q) = −(0.5 ln 0.25Same weights p, but the logs now come from q …
5. + 0.3 ln 0.25 + 0.2 ln 0.5)… one term per outcome …
6. ≈ 1.248… in nats.
7. KL = 1.248 − 1.030 ≈ 0.218KL(p ‖ q), the extra cost of using q.
8. 0.5 ln 2 ≈ 0.347Check with the definition Σ p_i ln(p_i/q_i), one term at a time …
9. 0.3 ln 1.2 ≈ 0.055… the second …
10. 0.2 ln 0.4 ≈ −0.183… and the third. This term is negative, where q gives more probability than p.
11. 0.347 + 0.055 − 0.183 ≈ 0.218The same KL. One term can be negative, but the total can't be.

Example 4: the KL divergence between two Gaussians

Find KL(p ‖ q) for p = N(1, 1) and q = N(0, 4), then the reverse direction.

1. μ₁ = 1, σ₁ = 1; μ₂ = 0, σ₂ = 2Read the standard deviations off the variances: √4 = 2.
2. ln(2/1) + (1 + 1²)/8 − 1/2ln(σ₂/σ₁) + (σ₁² + (μ₁ − μ₂)²)/(2σ₂²) − 1/2, with 2σ₂² = 2 · 4 = 8.
3. = 0.693 + 0.25 − 0.5 ≈ 0.443A narrow p sitting inside a wide q costs little.
4. KL(q ‖ p) = ln(1/2)Swap the roles: now σ₁ = 2, σ₂ = 1 …
5. + (4 + 1)/2 − 1/2… so the middle term is (4 + 1)/(2 · 1).
6. ≈ −0.693 + 2.5 − 0.5Evaluate each term.
7. KL(q ‖ p) ≈ 1.307A wide q has to be described by a narrow p, which is surprised by most of q's samples, so this direction is three times larger.

Common mistakes, and the exact line they happen on

1. Swapping the two arguments

KL(p ‖ q) for Example 3's p = (0.5, 0.3, 0.2) and q = (0.25, 0.25, 0.5).

1. KL(p ‖ q) = Σ p_i ln(p_i/q_i)Right so far.
2. Σ q_i ln(q_i/p_i) ≈ 0.239This is the mistake. This computed 0.25 ln(0.25/0.5) + 0.25 ln(0.25/0.3) + 0.5 ln(0.5/0.2). These are q's weights and q over p, which is KL(q ‖ p). The divergence isn't symmetric, so the two directions are different numbers: the right one is 0.218. The weights always come from the first argument, the distribution the outcomes are really drawn from. In a loss that is the data, which is why a classifier's loss has the labels as its first argument.

2. Mixing bits and nats in the perplexity

The perplexity of Example 1's source, with H = 1.5 bits.

1. H = 1.5 bitsRight so far.
2. perplexity = e^1.5 ≈ 4.48This is the mistake. The exponential's base must match the log used for H: 2^1.5 ≈ 2.83 for bits, or e^1.040 for nats, which is the same number. A perplexity of 4.48 for three outcomes is impossible; it can never exceed the number of outcomes.

3. Losing the minus sign

The entropy of Example 1 in nats.

1. Σ p_i ln p_i = 0.5 ln 0.5Right so far, apart from the sign out front …
2. + 0.25 ln 0.25… one term for each outcome …
3. + 0.25 ln 0.25… still with no minus in front.
4. H ≈ −1.040 natsThis is the mistake. Every log of a probability is ≤ 0, so the sum is negative and the definition puts a minus in front to make entropy ≥ 0. A negative entropy, or a negative cross-entropy loss, always means the sign was dropped. The right value is 1.040 nats.

Practice

1. Find the entropy of a fair eight-sided die in bits and in nats.

   Show answer

   log₂ 8 = 3 bits, which is 3 ln 2 ≈ 2.079 nats. A uniform distribution over K outcomes has entropy log K.

2. A classifier gives the true class probability 0.25. Find its cross-entropy loss on that example in nats and in bits.

   Show answer

   −ln 0.25 ≈ 1.386 nats, or −log₂ 0.25 = 2 bits.

3. Find KL(p ‖ q) for p = (1/2, 1/2) and q = (1/4, 3/4).

   Show answer

   (1/2) ln 2 + (1/2) ln(2/3) = (1/2) ln(4/3) ≈ 0.144.

4. A model's cross-entropy against the data is 1.5 nats, and the data's entropy is 1.2 nats. What is the KL divergence from the data to the model, and what is the lowest loss any model could reach?

   Show answer

   KL = 1.5 − 1.2 = 0.3 nats. No model can go below the entropy, 1.2 nats.

5. Find KL(N(2, 1) ‖ N(0, 1)).

   Show answer

   ln 1 + (1 + 4)/2 − 1/2 = 2. With equal variances it is (μ₁ − μ₂)²/(2σ²), which is symmetric in the means.