Entropy, cross-entropy and KL divergence

Entropy measures how uncertain a distribution is. Cross-entropy measures how surprised a model is by the real data, and KL divergence is the extra surprise that comes from the model being wrong.

Why it matters

A classifier outputs probabilities, and you need a loss that rewards putting high probability on the right answer and punishes confident mistakes hard. Cross-entropy is that loss, and nearly every classifier and every language model is trained on it. KL divergence measures how far apart two distributions are, and it shows up in VAEs, knowledge distillation and RLHF.

The idea

Entropy measures uncertainty, or 'surprise', in a distribution. Cross-entropy measures your surprise when you use your model q to predict data that really comes from p. KL divergence is the extra surprise: KL(p‖q) = cross-entropy − entropy.

Packing for a trip. With the true weather forecast (p), you pack exactly right. With a wrong forecast (q), you pack badly. KL is the extra, wasted luggage.

In the analogy, swapping the two forecasts feels like the same mistake. KL is not symmetric: KL(p‖q) is usually different from KL(q‖p).

How it works

  1. The surprise of an event with probability q is −log q. Certain events (q = 1) cost 0; rare ones cost a lot.
  2. Entropy is the average surprise under the true distribution: H(p) = −Σ p log p. A coin that lands heads 70% of the time has about 0.88 bits; a certain outcome has 0.
  3. Cross-entropy uses the model's probabilities for the surprise but the true distribution for the averaging: H(p, q) = −Σ p log q.
  4. With a one-hot true label, every term but one vanishes: the loss is −log of the probability the model gave the right class. For 0.7 that is 0.357.
  5. If the model had said 0.99, the loss would be about 0.01; if it had said 0.05, about 3.0. Confident mistakes are punished hardest.
  6. KL(p ‖ q) = H(p, q) − H(p). It is never negative, and it is 0 only when q = p. H(p) doesn't depend on the model, so minimising cross-entropy also minimises KL.

Key points

Common mistakes

Not true: KL divergence is a true distance: KL(p ‖ q) always equals KL(q ‖ p).
In fact: KL divergence is not symmetric, so it is not a distance. Swapping p and q usually changes its value.

Not true: Cross-entropy stays small when the model gives the true class a probability close to 0.
In fact: −log of a probability near 0 is huge, and −log 0 is infinite. A confident wrong answer is punished very heavily.

Go deeper: Why KL divergence can never be negative

  1. Start from one fact about the logarithm: log x ≤ x − 1 for every x > 0, with equality only at x = 1. (The straight line x − 1 touches the curve log x at x = 1 and lies above it everywhere else.)
  2. Write out the definition: KL(p ‖ q) = Σ p·log(p/q) = −Σ p·log(q/p).
  3. Apply the fact to each term, with x = q/p: log(q/p) ≤ q/p − 1. Multiply by p and add up: Σ p·log(q/p) ≤ Σ (q − p).
  4. Both distributions add up to 1, so Σ (q − p) = 1 − 1 = 0. Therefore Σ p·log(q/p) ≤ 0, which is the same as KL(p ‖ q) ≥ 0.
  5. Equality needs q/p = 1 in every term: that is, q = p.
  6. Since KL = H(p, q) − H(p), this also proves that cross-entropy is never below entropy. No model can be less surprised by the data than the true distribution is.

KL(p ‖ q) ≥ 0, with equality only when q = p. So H(p, q) ≥ H(p): the entropy of the data is the floor of the loss.

Part of Unit 1: Math foundations in the free ML Foundations course.