Mixed precision: fp16 and bf16
Mixed precision trains with 16-bit numbers (bf16 or fp16) for most of the maths while keeping a few sensitive values in 32-bit, roughly halving memory traffic and greatly speeding up training.
Why it matters
32-bit floats are more precise than neural networks need for most operations, and they cost twice the memory and bandwidth of 16-bit. GPUs' tensor cores are also far faster at 16-bit maths. Using 16-bit everywhere, though, can lose small updates or overflow. Mixed precision gets the speed while keeping training stable, and it is standard for every large model.
The idea
Mixed precision does most of the maths in 16-bit numbers instead of 32-bit: half the memory and much faster on modern GPUs. fp16 has more precision but a small range, so it can overflow. bf16 has the same range as fp32 with less precision, and it is the usual choice now.
Writing prices rounded to the nearest pound instead of exact to the penny. Faster, and usually good enough.
Some sums need the pennies. Sensitive parts, like the master copy of the weights and the optimiser update, stay in 32-bit.
How it works
- A float has a sign bit, exponent bits (which set the range) and mantissa bits (which set the precision).
- fp32: 8 exponent bits and 23 mantissa bits. fp16: 5 and 10. bf16: 8 and 7.
- fp16 has decent precision but a small range (largest value about 65,504), so gradients can overflow to inf or underflow to 0. It needs a gradient scaler: multiply the loss by a big number before backward, divide the gradients after.
- bf16 has fp32's range (up to about 3.4 × 10³⁸) with less precision, so it rarely overflows and usually needs no scaler. It is the common choice on modern GPUs.
- Under autocast, matrix multiplies and convolutions run in 16-bit; numerically sensitive operations such as reductions and softmax are kept in 32-bit.
- A 32-bit master copy of the weights receives the optimiser update, so tiny updates aren't rounded away (in bf16, 1 + 0.001 rounds back to 1).
Key points
- Exponent bits = range; mantissa bits = precision.
- bf16: fp32's range, low precision; fp16: more precision, small range.
- fp16 training needs loss scaling; bf16 usually doesn't.
- Keep fp32 master weights and do the optimiser step in fp32.
- 16-bit halves weight memory: 2 bytes per parameter instead of 4.
- In PyTorch, mixed precision is one line around the forward pass: torch.autocast(device_type="cuda", dtype=torch.bfloat16).
Common mistakes
- Training in fp16 without a gradient scaler, so small gradients round down to zero.
- Making the optimiser update in 16-bit, so small updates vanish.
- Comparing losses across precisions without expecting small differences.
- Assuming lower precision is always faster: older hardware may not support bf16 well.
Not true: 16-bit numbers: faster on old GPUs as well as on new ones.
In fact: Only on hardware built for it. Older GPUs may not support bf16 well, and then 16-bit is no faster, or even slower.
Not true: Optimiser step: done in 16-bit, to match the forward pass.
In fact: A tiny update added to a 16-bit weight is rounded away. Keep a 32-bit master copy of the weights and do the optimiser step in fp32.
Part of Unit 6: Systems and scale in the free ML Foundations course.