Parallelism: data, tensor and pipeline (the ideas)
Parallelism spreads training across many GPUs: data parallelism splits the batch, tensor parallelism splits each layer's matrices, pipeline parallelism splits the layers, and FSDP/ZeRO shard the training state.
Why it matters
Large models don't fit on one GPU, and even when they do, one GPU would take years to train them. Splitting the work is the only way to train at scale. How to split it depends on the model's size and on how fast the GPUs can talk to each other, and that choice decides how fast and how cheaply a model trains.
The idea
Data parallel: every GPU holds a full copy of the model, gets different data, and gradients are averaged. Tensor parallel: one layer's matrices are split across GPUs. Pipeline parallel: different layers live on different GPUs. FSDP and ZeRO split weights, gradients and optimiser states across GPUs to save memory.
Cooking a huge dinner. Data parallel: several kitchens cook the full menu for different tables. Tensor parallel: several chefs cut one giant cake together. Pipeline: an assembly line, one station per course.
Separate kitchens don't need to talk. GPUs talk constantly, and the speed of that communication often decides which method wins.
How it works
- Data parallel (DDP): every GPU holds the whole model and processes a different slice of the batch.
- After backward, an all-reduce averages the gradients across GPUs, so every copy takes the same step and stays identical.
- FSDP / ZeRO: like data parallel, but each GPU stores only a shard of the weights, gradients and optimiser states, gathering what it needs just in time. Much less memory per GPU.
- Tensor parallel: split each large matrix across GPUs; each computes part of every layer, and they exchange results inside each layer. Needs very fast links, so it is used within one machine.
- Pipeline parallel: put different layers on different GPUs and pass activations along, with micro-batches to keep stages busy. Idle time at the start and end is the "bubble".
- Large runs combine them ("3-D parallelism"): tensor parallel within a machine, pipeline across machines, data parallel across groups.
Key points
- Model fits on one GPU → data parallel; doesn't fit → FSDP, tensor or pipeline parallel.
- Communication speed often decides what works.
- All-reduce keeps data-parallel copies in sync.
- The slowest GPU sets the pace for everyone.
- Pipeline bubbles shrink with more micro-batches.
Common mistakes
- Using tensor parallelism across slow network links.
- Forgetting that the effective batch size grows with the number of data-parallel GPUs, so the learning rate may need adjusting.
- Not checking that the multi-GPU run matches a single-GPU run on a small test.
Not true: With eight data-parallel GPUs, the learning rate tuned for one GPU still fits unchanged.
In fact: Eight GPUs process eight times as many examples per step, so the effective batch is eight times larger. The learning rate usually has to be adjusted for it.
Not true: Tensor parallelism works well even across slow network links.
In fact: Tensor parallelism splits each layer across GPUs, so they exchange data inside every layer. It needs very fast links, such as GPUs in the same server.
Part of Unit 6: Systems and scale in the free ML Foundations course.