Self-attention: queries, keys and values

Self-attention lets every token look at all the tokens and build a new version of itself as a weighted mix of their values. The weights come from how well its query matches each token's key.

Why it matters

The meaning of a word depends on other words, often far away: "it" depends on what "it" refers to, and "bank" depends on "river" or "money". RNNs pass information step by step and forget. Attention lets every word look directly at every other word in one step, in parallel, and learn which ones matter. It is the core operation of every transformer.

The idea

Every token makes three vectors: a query (what I am looking for), a key (what I contain) and a value (what I will share). Each token compares its query with every key, turns the scores into weights with softmax, and takes a weighted average of the values.

A library. Your question is the query. Book titles are keys. The contents are values. You read a mix of the books whose titles best match your question.

In a library you read only the best books. In attention you get a blend of all of them, weighted by how well they match.

How it works

  1. Each token's vector x is multiplied by three learned matrices: query q = x·W_Q, key k = x·W_K, value v = x·W_V.
  2. For a token, compare its query with every key: score = q·k / √d. A high score means "this word is relevant to me".
  3. Softmax turns the scores into weights that add up to 1. For "it", most weight might go to "animal".
  4. The output for that token is the weighted sum of all the value vectors: mostly the animal's value, a little of the others.
  5. Do it for all tokens at once with matrices: A = softmax(QKᵀ/√d), output = A·V. A has shape (T, T): one row of weights per token.
  6. Divide by √d because dot products of long vectors are large; without it, softmax becomes nearly one-hot and gradients vanish.
  7. In a GPT, a causal mask sets the scores for future tokens to −∞ before the softmax, so each token only looks back.

Key points

Common mistakes

Not true: The causal mask can be applied after the softmax and still hide the future correctly.
In fact: After the softmax the future tokens have already taken their share of the weight, and the rows no longer add up to 1. Set their scores to −∞ before the softmax.

Not true: The √d scaling in attention is optional: leaving it out makes little difference.
In fact: Without dividing by √d, the scores are large and the softmax becomes almost one-hot. The gradients then vanish and training stalls.

Go deeper: Why the scores are divided by √d, and not by d

  1. Model a query q and a key k as vectors of d independent numbers, each with mean 0 and variance 1. That is roughly true at the start of training.
  2. The score is q·k = Σ qᵢkᵢ. Each term has mean 0 and variance E[qᵢ²]·E[kᵢ²] = 1.
  3. The d terms are independent, so their variances add: Var(q·k) = d. The typical size of a score is √d, about 23 for d = 512.
  4. Softmax compares scores through e to their difference. A score 22 above another gets about 3.6 billion times its weight, so one token takes almost everything and the gradient through the softmax is close to 0.
  5. Divide by √d: Var(q·k/√d) = d/d = 1, whatever the size of the model. The softmax sees scores of ordinary size and spreads its weight.
  6. Dividing by d would go too far: the variance would be 1/d, every score would be nearly equal, and attention would be an almost uniform average that ignores the content.

Var(q·k) = d, so q·k/√d has variance 1 for any model size: the softmax neither saturates nor goes flat.

Part of Unit 4: Transformers and LLMs in the free ML Foundations course.