Unit 6: Systems and scale
Make models fast and cheap. Understand what limits speed and memory on a GPU, and how training works across many GPUs.
The big picture
Same car, less fuel, more speed. Most of this unit is about moving data efficiently, not about doing more maths.
Lessons in this unit
- How a GPU works: compute vs memory: A GPU runs thousands of simple cores in parallel. How fast your code runs depends on whether it is limited by the maths (compute-bound) or by moving data to the cores (memory-bound).
- Mixed precision: fp16 and bf16: Mixed precision trains with 16-bit numbers (bf16 or fp16) for most of the maths while keeping a few sensitive values in 32-bit, roughly halving memory traffic and greatly speeding up training.
- Training memory: weights, gradients, optimiser states, activations: Training memory is the sum of the weights, the gradients, the optimiser states and the activations saved for backprop. With Adam and mixed precision, it is about 16 bytes per parameter before activations.
- Parallelism: data, tensor and pipeline (the ideas): Parallelism spreads training across many GPUs: data parallelism splits the batch, tensor parallelism splits each layer's matrices, pipeline parallelism splits the layers, and FSDP/ZeRO shard the training state.
- Inference speed: batching, quantisation, FlashAttention: Inference is sped up by serving many requests together (batching), storing weights in fewer bits (quantisation), and computing attention without writing the full score matrix to memory (FlashAttention).
- Profiling PyTorch code: Profiling measures where time is actually spent in your code — CPU, GPU, data loading, copies — so you optimise the real bottleneck instead of guessing.
- GPU kernels with Triton: A GPU kernel is a function that runs as many parallel instances on the GPU, and Triton lets you write kernels in Python-like code. The biggest win is usually fusion: doing several steps in one kernel, so that the data leaves slow memory only once.
- Pruning and distillation: Pruning removes weights, neurons or heads that matter little; distillation trains a small student model to imitate a large teacher's full output probabilities; both make models smaller and cheaper to run.
- Speculative decoding: Speculative decoding speeds up generation: a small draft model proposes several tokens, and the large model verifies them in a single pass. The output is exactly what the large model would have produced on its own.