Instruction tuning and preference tuning (RLHF, DPO)
Instruction tuning trains a pretrained model on example conversations so it follows requests, and preference tuning (RLHF or DPO) then trains it on pairs of answers so it prefers the better one.
Why it matters
A pretrained model is a brilliant autocomplete, not an assistant: ask it a question and it may continue with more questions. Instruction tuning teaches the format and habit of helpful answering; preference tuning sharpens qualities that are easy to judge but hard to demonstrate, like helpfulness, honesty and harmlessness. This is the step that turned GPT-3 into ChatGPT.
The idea
A pretrained model only continues text. Instruction tuning (supervised fine-tuning) trains it on examples of good question-and-answer pairs. Preference tuning trains it on pairs of answers where people marked the better one. RLHF uses a reward model and reinforcement learning. DPO learns from the pairs directly.
Pretraining is reading everything. Instruction tuning is an internship with worked examples. Preference tuning is a manager saying 'this version is better than that one'.
A manager explains why. Preference data only says which answer won, so models can learn shortcuts, such as 'longer answers win'.
How it works
- Supervised fine-tuning (SFT): train on thousands of (prompt, ideal answer) conversations, written in a chat template, with the loss computed only on the answer tokens.
- Collect preferences: for one prompt, sample two answers and have people (or a strong model) mark the better one: (prompt, chosen, rejected).
- RLHF, step 1: train a reward model to give the chosen answer a higher score than the rejected one.
- RLHF, step 2: use reinforcement learning (often PPO) to make the model produce high-reward answers. A KL penalty keeps it close to the SFT model, so it doesn't game the reward.
- DPO skips both the reward model and RL: a classification-style loss directly makes the chosen answer more likely than the rejected one, relative to a frozen reference model.
- Evaluate carefully: preference tuning can teach shortcuts, such as longer or more flattering answers winning.
Key points
- Pretraining → knowledge; SFT → format and instruction following; preference tuning → quality and safety.
- RLHF = reward model + RL (PPO) + KL penalty.
- DPO = learn directly from preference pairs; simpler and more stable.
- Reward hacking: the model learns what the reward likes, not what people want.
- Chat templates (special tokens marking roles) must match between training and use.
Common mistakes
- Computing the SFT loss on the prompt tokens as well as the answers.
- Preference data with hidden biases, such as "longer is always better".
- Dropping the KL penalty, so the model drifts into strange, reward-gaming text.
Not true: SFT loss: computed on the prompt tokens as well as on the answer.
In fact: The model should learn to write answers, not to predict what users will ask. The loss is taken on the answer tokens only.
Not true: KL penalty in RLHF: an optional extra that can be dropped.
In fact: The KL penalty keeps the model close to the SFT model. Without it, the model drifts into strange text that pleases the reward model but not people.
Part of Unit 4: Transformers and LLMs in the free ML Foundations course.