Unit 4: Transformers and LLMs
Build a small GPT from scratch: tokenizer, attention, training and fine-tuning. This is how modern AI works inside.
The big picture
An LLM is a deep neural network with one special layer (attention), trained to guess the next token on a huge amount of text. Everything else is detail, but important detail.
Lessons in this unit
- Tokenisation and byte-pair encoding (BPE): Tokenisation splits text into the units a language model reads, called tokens. Byte-pair encoding (BPE) builds those units by starting from single characters (or bytes) and repeatedly merging the most frequent neighbouring pair.
- Token embeddings and positional encoding: Token embeddings turn each token id into a learned vector, and positional encodings add information about where each token sits, because attention on its own ignores order.
- Self-attention: queries, keys and values: Self-attention lets every token look at all the tokens and build a new version of itself as a weighted mix of their values. The weights come from how well its query matches each token's key.
- Multi-head attention and the transformer block: Multi-head attention runs several smaller attentions side by side and joins their results. A transformer block wraps attention and an MLP, each with a layer norm and a residual connection, and a model stacks many of these blocks.
- GPT-style models and next-token prediction: A GPT is a stack of transformer blocks with a causal mask, trained to predict the next token; it generates text by predicting a token, appending it and repeating.
- Encoder models (BERT) vs decoder models (GPT): Encoder models, like BERT, let every token look in both directions and learn by filling in masked words, which suits understanding text. Decoder models, like GPT, look only backwards and learn by predicting the next word, which suits writing it.
- Vision transformers and CLIP: A vision transformer (ViT) cuts an image into patches and feeds them to a standard transformer as if they were tokens. CLIP trains an image encoder and a text encoder together, so that a picture and its caption get similar vectors.
- Sampling: greedy, temperature, top-k, top-p: Decoding turns the model's next-token probabilities into an actual choice: greedy takes the top token, temperature reshapes the distribution, and top-k and top-p cut off unlikely tokens before sampling.
- KV cache and the cost of generation: The KV cache stores the keys and values of every token already processed. Generating a new token then computes only that token's keys and values, instead of recomputing them for the whole sequence.
- Pretraining data and scaling laws: Pretraining is next-token prediction on an enormous text collection, and scaling laws describe how the resulting loss falls smoothly and predictably as model size, data and compute grow.
- Mixture of experts: A mixture-of-experts (MoE) layer replaces one big MLP with many expert MLPs, plus a router that sends each token to only a few of them. The model has many parameters, but uses only a fraction of them for each token.
- Fine-tuning and LoRA: Fine-tuning continues training a pretrained model on your own data. LoRA does it cheaply: it freezes the original weights W and trains a small low-rank update, so the layer uses W + B·A.
- Instruction tuning and preference tuning (RLHF, DPO): Instruction tuning trains a pretrained model on example conversations so it follows requests, and preference tuning (RLHF or DPO) then trains it on pairs of answers so it prefers the better one.
- Reasoning models and test-time compute: A reasoning model is trained to write out its steps before answering. Test-time compute means spending more computation at answer time, through longer reasoning or several samples and a vote, to get better answers.