Multi-head attention and the transformer block

Multi-head attention runs several smaller attentions side by side and joins their results. A transformer block wraps attention and an MLP, each with a layer norm and a residual connection, and a model stacks many of these blocks.

Why it matters

One attention pattern per layer is limiting: a word may need to look at its subject, its object and the previous word all at once. Several heads can each specialise. And attention alone only mixes information between tokens; the MLP then processes it inside each token. The block is the repeating unit of every transformer, so knowing it means you can read any model's architecture.

The idea

Multi-head attention runs several smaller attentions in parallel, so each head can track a different kind of relationship. A transformer block is attention plus an MLP, each wrapped with layer norm and a residual connection.

A team reading the same contract. One person checks dates, one checks names, one checks money. Then they merge their notes.

People choose their roles. Heads learn their roles on their own, and many heads do not have clean, readable roles.

How it works

  1. Split d_model into h heads: with d_model = 768 and 12 heads, each head works with 64-dimensional queries, keys and values.
  2. Each head computes its own attention, with its own projection matrices, so each can learn a different relationship.
  3. Join the heads' outputs back together (768 numbers per token) and mix them with one more matrix, W_O.
  4. Pre-norm block: x = x + Attention(LayerNorm(x)). Attention lets tokens exchange information.
  5. Then x = x + MLP(LayerNorm(x)). The MLP is usually 4 times wider inside (768 → 3,072 → 768) and works on each token on its own.
  6. Stack N identical blocks (12 in GPT-2 small, dozens in large models). Every block reads from and writes to the residual stream.
  7. After the last block: a final layer norm, then a linear layer to vocabulary-sized logits.

Key points

Common mistakes

Not true: A deep stack of transformer blocks trains equally well without residual connections.
In fact: Without the residual path, gradients fade through a deep stack and training gets much worse. Every block adds its output back to its input.

Not true: Eight attention heads cost about eight times as much as one big head.
In fact: Each head works with vectors of size d/heads, so the total cost is about the same as one big head. The work is split up, not multiplied.

Part of Unit 4: Transformers and LLMs in the free ML Foundations course.