Fine-tuning and LoRA
Fine-tuning continues training a pretrained model on your own data. LoRA does it cheaply: it freezes the original weights W and trains a small low-rank update, so the layer uses W + B·A.
Why it matters
Pretrained models are general; you often need a specific style, format or domain. Full fine-tuning updates billions of weights, needs optimiser memory for all of them, and produces a whole new model copy per task. LoRA trains well under 1% of the parameters, fits on a single GPU, and produces small adapters you can swap in and out of one base model.
The idea
Fine-tuning continues training a pretrained model on your own data. Full fine-tuning updates every weight. LoRA freezes the original weights W and learns a small update BA of low rank r (for example 8): W′ = W + BA.
Instead of rewriting a whole textbook, you add a few sticky notes with corrections.
Sticky notes can be removed, and so can LoRA adapters, so one base model can serve many tasks. But a small rank cannot teach big new skills.
How it works
- Pick the weight matrices to adapt, usually the attention projections (and often the MLP).
- Freeze W. Add two small matrices: B (d_out × r) and A (r × d_in), with rank r such as 8 or 16.
- Start B at zero, so B·A = 0 and the model begins exactly as the pretrained one.
- The layer computes y = x·Wᵀ + x·(B·A)ᵀ, usually scaled by α/r.
- Train only A and B. For a 4,096 × 4,096 matrix with r = 8, that is 65,536 weights instead of 16.8 million: about 0.4%.
- Afterwards, keep the adapter separate (tiny files, swappable), or merge it: W′ = W + B·A, with no extra cost when running.
- QLoRA goes further: the frozen base model is stored in 4-bit, so even large models can be fine-tuned on one GPU.
Key points
- Full fine-tuning: every weight, most flexible, most expensive.
- LoRA: frozen W plus a trainable low-rank B·A.
- Trainable LoRA parameters per matrix = r × (d_in + d_out).
- B = 0 at the start means no change at step 0.
- Adapters can be merged into W for free inference, or swapped per task.
- Fine-tuning teaches style and format well; it is a poor way to add lots of new facts (use RAG, Unit 5).
Common mistakes
- Starting both A and B random, so the model is damaged at step 0.
- Choosing a rank far too small for a big change in behaviour, or far too big for a small dataset.
- Fine-tuning to inject facts that change often.
- Forgetting the original chat template, so the tuned model loses its formatting.
Not true: Facts that change each week: add them by fine-tuning.
In fact: Retraining for each change is slow and costly, and the old facts stay in the weights. Facts that change belong in retrieval (RAG), not in fine-tuning.
Not true: LoRA rank: r = 1 covers a big change in behaviour as well as a small one.
In fact: The rank limits how much the update can express. A big change in behaviour needs a higher rank; a small dataset needs a lower one, or it overfits.
Part of Unit 4: Transformers and LLMs in the free ML Foundations course.