Softmax and cross-entropy loss
Softmax turns a vector of raw scores (logits) into probabilities that sum to 1. Cross-entropy loss is −log of the probability given to the correct class. Together, their gradient is simply p − y.
Why it matters
A classifier's last layer outputs arbitrary numbers, but you want probabilities, and a loss that strongly punishes confident wrong answers. Softmax plus cross-entropy is the standard pairing for every multi-class classifier and every language model, which predicts the next token from a softmax over tens of thousands of options.
The idea
Softmax turns raw scores (logits) into probabilities that sum to 1: exp(zᵢ) / Σ exp(zⱼ). Cross-entropy loss is −log(probability of the correct class). Together, their gradient is simply: probabilities − one-hot label.
Turning vote counts into vote shares.
Real vote shares are proportional. Softmax is exponential, so a small lead in score becomes a big lead in probability.
How it works
- Start with logits, one per class: z = [2, 1, 0.1].
- Exponentiate: e^z = [7.39, 2.72, 1.11]. All positive, and the leader pulls further ahead.
- Divide by the total (11.21): p = [0.66, 0.24, 0.10]. They add up to 1.
- For stability, subtract the largest logit first. The answer is identical, but exp never overflows.
- If the true class is the first one, the loss is −log 0.66 ≈ 0.42.
- The gradient with respect to the logits is p − y = [0.66 − 1, 0.24, 0.10] = [−0.34, 0.24, 0.10]: raise the right score, lower the others in proportion.
Key points
- Softmax turns scores into probabilities: softmax(z)ᵢ = exp(zᵢ) / Σⱼ exp(zⱼ). Each one is positive, and together they add up to 1.
- Adding the same number to every logit doesn't change the softmax, so subtracting the max is free and safe.
- Loss = −log p(correct class). Its gradient on the logits is p − one_hot(y).
- In PyTorch, F.cross_entropy takes raw logits and applies log-softmax itself. Don't softmax first.
- Scaling logits by 1/T (temperature) makes the distribution sharper (T < 1) or flatter (T > 1). Used in sampling (Unit 4).
- At random initialisation with K classes, the loss starts near ln K.
Common mistakes
- Applying softmax and then F.cross_entropy, which applies it again.
- Computing exp of large logits without subtracting the max: overflow, inf, NaN.
- Taking log(softmax) in two steps instead of using log-softmax, which loses precision.
Not true: The model should apply softmax to its outputs before passing them to F.cross_entropy.
In fact: F.cross_entropy applies log-softmax itself, so it expects raw logits. With a softmax in front, it is applied twice and the loss is wrong.
Not true: exp of the logits can be computed directly: large logits cause no trouble.
In fact: exp of a large logit overflows to inf, and the result becomes NaN. Subtract the largest logit first: the softmax does not change, and nothing overflows.
Part of Unit 3: Deep learning foundations in the free ML Foundations course.