Reasoning models and test-time compute

A reasoning model is trained to write out its steps before answering. Test-time compute means spending more computation at answer time, through longer reasoning or several samples and a vote, to get better answers.

Why it matters

Some problems can't be solved in a single forward pass: multi-step maths, tricky code, planning. Letting the model work step by step, and spend more tokens on harder questions, greatly improves accuracy. It gives a new knob — accuracy versus cost per query — alongside model size, and it is behind the jump in maths and coding ability of recent models.

The idea

A reasoning model is trained to 'think' before answering: it writes out intermediate steps (a chain of thought) and can spend more tokens on harder problems.

Spending more computation at answer time, by reasoning for longer or by sampling several answers and picking the best, is called test-time compute. Much of this ability is trained with reinforcement learning on problems whose answers can be checked, like maths and code.

A student allowed to use scrap paper and check their work, instead of shouting the first answer that comes to mind.

A student's scrap paper shows their real reasoning. A model's written steps are not guaranteed to be the true cause of its answer. And more thinking costs more time and money; it helps most on problems with checkable steps.

How it works

  1. Chain of thought: the model writes its reasoning first ("12 pens is 4 groups of 3 …"), then the answer. Each step is a smaller, easier prediction that the next step can build on.
  2. Prompting ("think step by step") helps a little; training helps a lot. Reasoning models are trained with reinforcement learning on problems whose answers can be checked, such as maths with known answers and code with unit tests.
  3. The reward is simply "was the final answer right?" Over many attempts, the model learns reasoning habits that lead to right answers: checking work, trying again, breaking problems down.
  4. Self-consistency: sample several reasoning paths at a temperature above 0 and take a majority vote on the final answers.
  5. Best-of-N with a verifier: generate N answers and let a checker (tests, a reward model) choose.
  6. More thinking costs more tokens and time, so it is worth it for hard, checkable problems, not for simple lookups.

Key points

Common mistakes

Not true: The reasoning a model writes out is a faithful record of how it reached its answer.
In fact: The written steps can leave out or misstate what really drove the answer. Treat them as useful output, not as a guaranteed explanation.

Not true: Self-consistency works at temperature 0.
In fact: At temperature 0 every sample is the same answer, so there is nothing to vote on. Self-consistency needs varied attempts, so sample with some randomness.

Part of Unit 4: Transformers and LLMs in the free ML Foundations course.