Parallelism: data, tensor and pipeline (the ideas)

Parallelism spreads training across many GPUs: data parallelism splits the batch, tensor parallelism splits each layer's matrices, pipeline parallelism splits the layers, and FSDP/ZeRO shard the training state.

Why it matters

Large models don't fit on one GPU, and even when they do, one GPU would take years to train them. Splitting the work is the only way to train at scale. How to split it depends on the model's size and on how fast the GPUs can talk to each other, and that choice decides how fast and how cheaply a model trains.

The idea

Data parallel: every GPU holds a full copy of the model, gets different data, and gradients are averaged. Tensor parallel: one layer's matrices are split across GPUs. Pipeline parallel: different layers live on different GPUs. FSDP and ZeRO split weights, gradients and optimiser states across GPUs to save memory.

Cooking a huge dinner. Data parallel: several kitchens cook the full menu for different tables. Tensor parallel: several chefs cut one giant cake together. Pipeline: an assembly line, one station per course.

Separate kitchens don't need to talk. GPUs talk constantly, and the speed of that communication often decides which method wins.

How it works

  1. Data parallel (DDP): every GPU holds the whole model and processes a different slice of the batch.
  2. After backward, an all-reduce averages the gradients across GPUs, so every copy takes the same step and stays identical.
  3. FSDP / ZeRO: like data parallel, but each GPU stores only a shard of the weights, gradients and optimiser states, gathering what it needs just in time. Much less memory per GPU.
  4. Tensor parallel: split each large matrix across GPUs; each computes part of every layer, and they exchange results inside each layer. Needs very fast links, so it is used within one machine.
  5. Pipeline parallel: put different layers on different GPUs and pass activations along, with micro-batches to keep stages busy. Idle time at the start and end is the "bubble".
  6. Large runs combine them ("3-D parallelism"): tensor parallel within a machine, pipeline across machines, data parallel across groups.

Key points

Common mistakes

Not true: With eight data-parallel GPUs, the learning rate tuned for one GPU still fits unchanged.
In fact: Eight GPUs process eight times as many examples per step, so the effective batch is eight times larger. The learning rate usually has to be adjusted for it.

Not true: Tensor parallelism works well even across slow network links.
In fact: Tensor parallelism splits each layer across GPUs, so they exchange data inside every layer. It needs very fast links, such as GPUs in the same server.

Part of Unit 6: Systems and scale in the free ML Foundations course.