Sampling: greedy, temperature, top-k, top-p

Decoding turns the model's next-token probabilities into an actual choice: greedy takes the top token, temperature reshapes the distribution, and top-k and top-p cut off unlikely tokens before sampling.

Why it matters

The same model can sound dull and repetitive or creative and unhinged depending only on how you pick tokens. Greedy decoding tends to loop; pure sampling sometimes picks absurd words from the long tail. Decoding settings are the cheapest, most-used control you have over an LLM's output.

The idea

The model gives a probability for every possible next token. Greedy decoding always picks the top one. Temperature divides the logits by T before softmax: T < 1 makes choices sharper, T > 1 makes them flatter. Top-k keeps only the k best tokens. Top-p keeps the smallest set whose probabilities add up to p.

Ordering at a restaurant. Greedy: always the most popular dish. High temperature: feeling adventurous. Top-k: only choose from the top 5 dishes.

One restaurant order does not change the next one. In text, each choice changes everything after it, so one odd pick can send the text off track.

How it works

  1. The model gives logits for every token in the vocabulary. Softmax turns them into probabilities.
  2. Greedy: always take the most likely token. Deterministic, but prone to repetition.
  3. Temperature T: divide the logits by T before the softmax. T < 1 sharpens the distribution (safer); T > 1 flattens it (more varied); T → 0 becomes greedy.
  4. Top-k: keep only the k most likely tokens, set the rest to zero, renormalise, then sample.
  5. Top-p (nucleus): sort tokens by probability and keep the smallest set whose probabilities add up to at least p (such as 0.9). The set is small when the model is sure and larger when it isn't.
  6. Sample from what is left. A fixed random seed makes sampling repeatable.
  7. Typical settings: T around 0.7 with top-p 0.9 for chat; T = 0 for extraction, classification and code that must be exact.

Key points

Common mistakes

Not true: Temperature 0 = the setting for creative writing.
In fact: At temperature 0 the model always takes the most likely token, which makes the text repetitive and can trap it in loops. Creative writing needs some randomness.

Not true: High temperature: the choice for answers that have to be exact and repeatable.
In fact: High temperature adds randomness, so answers vary and errors appear. For exact, repeatable output use a low temperature, or greedy decoding.

Part of Unit 4: Transformers and LLMs in the free ML Foundations course.