Mixed precision: fp16 and bf16

Mixed precision trains with 16-bit numbers (bf16 or fp16) for most of the maths while keeping a few sensitive values in 32-bit, roughly halving memory traffic and greatly speeding up training.

Why it matters

32-bit floats are more precise than neural networks need for most operations, and they cost twice the memory and bandwidth of 16-bit. GPUs' tensor cores are also far faster at 16-bit maths. Using 16-bit everywhere, though, can lose small updates or overflow. Mixed precision gets the speed while keeping training stable, and it is standard for every large model.

The idea

Mixed precision does most of the maths in 16-bit numbers instead of 32-bit: half the memory and much faster on modern GPUs. fp16 has more precision but a small range, so it can overflow. bf16 has the same range as fp32 with less precision, and it is the usual choice now.

Writing prices rounded to the nearest pound instead of exact to the penny. Faster, and usually good enough.

Some sums need the pennies. Sensitive parts, like the master copy of the weights and the optimiser update, stay in 32-bit.

How it works

  1. A float has a sign bit, exponent bits (which set the range) and mantissa bits (which set the precision).
  2. fp32: 8 exponent bits and 23 mantissa bits. fp16: 5 and 10. bf16: 8 and 7.
  3. fp16 has decent precision but a small range (largest value about 65,504), so gradients can overflow to inf or underflow to 0. It needs a gradient scaler: multiply the loss by a big number before backward, divide the gradients after.
  4. bf16 has fp32's range (up to about 3.4 × 10³⁸) with less precision, so it rarely overflows and usually needs no scaler. It is the common choice on modern GPUs.
  5. Under autocast, matrix multiplies and convolutions run in 16-bit; numerically sensitive operations such as reductions and softmax are kept in 32-bit.
  6. A 32-bit master copy of the weights receives the optimiser update, so tiny updates aren't rounded away (in bf16, 1 + 0.001 rounds back to 1).

Key points

Common mistakes

Not true: 16-bit numbers: faster on old GPUs as well as on new ones.
In fact: Only on hardware built for it. Older GPUs may not support bf16 well, and then 16-bit is no faster, or even slower.

Not true: Optimiser step: done in 16-bit, to match the forward pass.
In fact: A tiny update added to a 16-bit weight is rounded away. Keep a 32-bit master copy of the weights and do the optimiser step in fp32.

Part of Unit 6: Systems and scale in the free ML Foundations course.