Inference speed: batching, quantisation, FlashAttention

Inference is sped up by serving many requests together (batching), storing weights in fewer bits (quantisation), and computing attention without writing the full score matrix to memory (FlashAttention).

Why it matters

Generating text is mostly memory-bound: every token requires reading all the weights and the KV cache. Serving cost and speed decide whether a model is affordable to deploy. These three techniques are the main reasons modern models can be served cheaply and quickly, and on smaller hardware.

The idea

Batching serves many requests together to use the GPU fully. Quantisation stores weights in 8 or 4 bits to save memory and bandwidth. FlashAttention computes attention in small tiles that stay in fast on-chip memory, so the big T × T matrix is never written out.

Batching is a bus instead of many taxis. Quantisation is a compressed photo: smaller, and usually looks the same.

A bus waits to fill up, so the first passenger waits longer. Heavy compression shows artefacts: at 4 bits or fewer, quality can drop, so always measure it.

How it works

  1. Batching: one pass over the weights serves many users' next tokens at once, so the cost of reading the weights is shared. Continuous batching adds and removes requests while the batch is running.
  2. Quantisation: store weights as 8-bit or 4-bit integers plus a scale factor, instead of 16-bit floats.
  3. Each weight snaps to its nearest allowed level. Per-channel or per-group scales keep errors small; an outlier weight in a group stretches the scale and hurts everyone else in it.
  4. Smaller weights mean less memory and fewer bytes per token: a 70B model needs about 140 GB in bf16, 70 GB in int8 and 35 GB in 4-bit.
  5. FlashAttention splits attention into tiles that fit in fast on-chip memory, computes the softmax piece by piece with running totals, and never writes the T × T matrix to slow memory.
  6. The result is identical attention with far less memory traffic, and memory that grows linearly with sequence length.

Key points

Common mistakes

Not true: FlashAttention: faster, at the price of slightly different outputs.
In fact: It computes exactly the same result. It only changes how the work is arranged in memory, so that less data has to move.

Not true: Quantisation: one scale for a whole tensor works well even when it contains outliers.
In fact: One huge value stretches the scale, so the many small values round to the same few numbers. Finer-grained scales, per channel or per group, fix that.

Part of Unit 6: Systems and scale in the free ML Foundations course.