Training memory: weights, gradients, optimiser states, activations

Training memory is the sum of the weights, the gradients, the optimiser states and the activations saved for backprop. With Adam and mixed precision, it is about 16 bytes per parameter before activations.

Why it matters

Running out of GPU memory is the most common reason a training run won't start. Knowing where the bytes go lets you estimate whether a model fits before trying, and pick the right fix — a smaller batch, checkpointing, an 8-bit optimiser, or sharding across GPUs.

The idea

Training needs memory for the weights, the gradients, the optimiser states (Adam keeps 2 extra numbers per weight) and the activations saved for backprop. With mixed precision and Adam, a common estimate is about 16 bytes per parameter, before activations.

Moving house. It is not just the furniture (weights). You also need boxes (gradients), labels (optimiser states) and floor space to lay things out while packing (activations).

Moving house happens once. Activations grow with batch size and sequence length, so the same model can fit or not fit depending on your batch.

How it works

  1. Weights in bf16: 2 bytes per parameter.
  2. Gradients in bf16: another 2 bytes per parameter.
  3. An fp32 master copy of the weights for the update: 4 bytes.
  4. Adam's two running averages (m and v), each fp32: 8 bytes.
  5. Total ≈ 16 bytes per parameter. A 1-billion-parameter model needs about 16 GB before activations, compared with about 2 GB just to run it in bf16.
  6. Activations: every layer's outputs are saved for backprop. They grow with batch size × sequence length × width × layers, and often dominate for long sequences.
  7. Fixes: gradient checkpointing (store fewer activations, recompute them in backward), gradient accumulation (smaller micro-batches), 8-bit optimisers, LoRA (far fewer trainable parameters), and ZeRO/FSDP (split weights, gradients and optimiser states across GPUs).

Key points

Common mistakes

Not true: Halving the batch size saves memory and has no other effect on training.
In fact: A smaller batch gives noisier gradients, so training behaves differently. To keep the same effective batch, add up the gradients of several small batches before each update.

Not true: Training memory can be estimated from the size of the weights alone.
In fact: Training also stores gradients, optimiser states and the activations saved for backprop. With Adam that is about 16 bytes per parameter, before activations.

Part of Unit 6: Systems and scale in the free ML Foundations course.