Speculative decoding

Speculative decoding speeds up generation: a small draft model proposes several tokens, and the large model verifies them in a single pass. The output is exactly what the large model would have produced on its own.

Why it matters

Large models generate one token per forward pass, and each pass is slow because it is memory-bound: the GPU spends its time reading weights, not computing. Checking several proposed tokens costs about the same as generating one. Speculative decoding turns that spare capacity into speed, with no change in output quality.

The idea

A big model generates one token at a time, and each step is slow. In speculative decoding, a small, fast 'draft' model guesses several tokens ahead. The big model then checks all the guesses in one pass, keeps the ones it agrees with, and replaces the first wrong one. The output is exactly what the big model alone would produce, only faster.

A junior writer drafts the next sentence, and a senior editor checks it in one read, instead of the senior writing every word slowly.

An editor can rewrite anything. Here, only the guesses up to the first rejected token are kept, so if the draft model guesses badly you gain almost nothing. The speed-up depends on the acceptance rate.

How it works

  1. A small, fast draft model (or extra prediction heads) proposes the next k tokens, for example 4.
  2. The large target model runs once over the context plus all k proposals, producing its own prediction at every position in parallel.
  3. Walk through the proposals: accept each one the target model agrees with (with a probabilistic acceptance rule when sampling).
  4. At the first disagreement, discard the rest and take the target model's own token there instead.
  5. So each expensive pass yields between 1 and k + 1 tokens; with 3 of 4 accepted, you get 4 tokens from one pass.
  6. The acceptance rule guarantees the output has exactly the target model's distribution: speed without any quality change.

Key points

Common mistakes

Not true: Speculative decoding gives its largest speed-up at very large batch sizes.
In fact: At large batch sizes the GPU is already busy, so the spare capacity that speculative decoding uses is gone. The gains are largest for single requests and small batches.

Not true: Speculative decoding lowers the quality of the output a little.
In fact: The large model checks every drafted token and keeps only what it would have produced itself. The output is exactly the same; only the speed changes.

Part of Unit 6: Systems and scale in the free ML Foundations course.