Self-attention: queries, keys and values
Self-attention lets every token look at all the tokens and build a new version of itself as a weighted mix of their values. The weights come from how well its query matches each token's key.
Why it matters
The meaning of a word depends on other words, often far away: "it" depends on what "it" refers to, and "bank" depends on "river" or "money". RNNs pass information step by step and forget. Attention lets every word look directly at every other word in one step, in parallel, and learn which ones matter. It is the core operation of every transformer.
The idea
Every token makes three vectors: a query (what I am looking for), a key (what I contain) and a value (what I will share). Each token compares its query with every key, turns the scores into weights with softmax, and takes a weighted average of the values.
A library. Your question is the query. Book titles are keys. The contents are values. You read a mix of the books whose titles best match your question.
In a library you read only the best books. In attention you get a blend of all of them, weighted by how well they match.
How it works
- Each token's vector x is multiplied by three learned matrices: query q = x·W_Q, key k = x·W_K, value v = x·W_V.
- For a token, compare its query with every key: score = q·k / √d. A high score means "this word is relevant to me".
- Softmax turns the scores into weights that add up to 1. For "it", most weight might go to "animal".
- The output for that token is the weighted sum of all the value vectors: mostly the animal's value, a little of the others.
- Do it for all tokens at once with matrices: A = softmax(QKᵀ/√d), output = A·V. A has shape (T, T): one row of weights per token.
- Divide by √d because dot products of long vectors are large; without it, softmax becomes nearly one-hot and gradients vanish.
- In a GPT, a causal mask sets the scores for future tokens to −∞ before the softmax, so each token only looks back.
Key points
- The whole operation in one line: Attention(Q, K, V) = softmax(QKᵀ/√d)·V. Score, scale, turn into weights, then mix the values.
- Query: what I am looking for. Key: what I offer for matching. Value: what I pass on.
- Every pair of tokens is compared, so the cost grows with the square of the sequence length.
- The weights depend on the input: attention is a data-dependent mixing of tokens.
- All tokens are processed in parallel, unlike an RNN.
- Self-attention: Q, K and V come from the same sequence. Cross-attention: queries from one sequence, keys and values from another.
Common mistakes
- Forgetting the √d scaling, so the scores are large, the softmax becomes almost one-hot and gradients vanish.
- Applying the softmax over the wrong axis: each row (one query) must sum to 1.
- Adding the causal mask after the softmax instead of before.
- Mixing up the roles of keys (for matching) and values (for the content passed on).
Not true: The causal mask can be applied after the softmax and still hide the future correctly.
In fact: After the softmax the future tokens have already taken their share of the weight, and the rows no longer add up to 1. Set their scores to −∞ before the softmax.
Not true: The √d scaling in attention is optional: leaving it out makes little difference.
In fact: Without dividing by √d, the scores are large and the softmax becomes almost one-hot. The gradients then vanish and training stalls.
Go deeper: Why the scores are divided by √d, and not by d
- Model a query q and a key k as vectors of d independent numbers, each with mean 0 and variance 1. That is roughly true at the start of training.
- The score is q·k = Σ qᵢkᵢ. Each term has mean 0 and variance E[qᵢ²]·E[kᵢ²] = 1.
- The d terms are independent, so their variances add: Var(q·k) = d. The typical size of a score is √d, about 23 for d = 512.
- Softmax compares scores through e to their difference. A score 22 above another gets about 3.6 billion times its weight, so one token takes almost everything and the gradient through the softmax is close to 0.
- Divide by √d: Var(q·k/√d) = d/d = 1, whatever the size of the model. The softmax sees scores of ordinary size and spreads its weight.
- Dividing by d would go too far: the variance would be 1/d, every score would be nearly equal, and attention would be an almost uniform average that ignores the content.
Var(q·k) = d, so q·k/√d has variance 1 for any model size: the softmax neither saturates nor goes flat.
Part of Unit 4: Transformers and LLMs in the free ML Foundations course.