Softmax and cross-entropy loss

Softmax turns a vector of raw scores (logits) into probabilities that sum to 1. Cross-entropy loss is −log of the probability given to the correct class. Together, their gradient is simply p − y.

Why it matters

A classifier's last layer outputs arbitrary numbers, but you want probabilities, and a loss that strongly punishes confident wrong answers. Softmax plus cross-entropy is the standard pairing for every multi-class classifier and every language model, which predicts the next token from a softmax over tens of thousands of options.

The idea

Softmax turns raw scores (logits) into probabilities that sum to 1: exp(zᵢ) / Σ exp(zⱼ). Cross-entropy loss is −log(probability of the correct class). Together, their gradient is simply: probabilities − one-hot label.

Turning vote counts into vote shares.

Real vote shares are proportional. Softmax is exponential, so a small lead in score becomes a big lead in probability.

How it works

  1. Start with logits, one per class: z = [2, 1, 0.1].
  2. Exponentiate: e^z = [7.39, 2.72, 1.11]. All positive, and the leader pulls further ahead.
  3. Divide by the total (11.21): p = [0.66, 0.24, 0.10]. They add up to 1.
  4. For stability, subtract the largest logit first. The answer is identical, but exp never overflows.
  5. If the true class is the first one, the loss is −log 0.66 ≈ 0.42.
  6. The gradient with respect to the logits is p − y = [0.66 − 1, 0.24, 0.10] = [−0.34, 0.24, 0.10]: raise the right score, lower the others in proportion.

Key points

Common mistakes

Not true: The model should apply softmax to its outputs before passing them to F.cross_entropy.
In fact: F.cross_entropy applies log-softmax itself, so it expects raw logits. With a softmax in front, it is applied twice and the loss is wrong.

Not true: exp of the logits can be computed directly: large logits cause no trouble.
In fact: exp of a large logit overflows to inf, and the result becomes NaN. Subtract the largest logit first: the softmax does not change, and nothing overflows.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.