Pruning and distillation
Pruning removes weights, neurons or heads that matter little; distillation trains a small student model to imitate a large teacher's full output probabilities; both make models smaller and cheaper to run.
Why it matters
The best models are often too large, slow or expensive for phones, high-traffic services or tight latency budgets. Many of their parameters contribute little, and much of a big model's knowledge can be transferred into a smaller one. Pruning and distillation, alongside quantisation, are the standard ways to shrink models for deployment.
The idea
Two ways to make a model smaller. Pruning removes weights, neurons or attention heads that matter little. Distillation trains a small 'student' model to copy a big 'teacher' model's full output probabilities, not just its top answer.
Pruning is trimming a tree's weak branches. Distillation is an expert teacher who explains not only the right answer, but also how close each wrong answer was.
A trimmed tree regrows on its own. A pruned network usually needs retraining to recover accuracy. And removing scattered single weights rarely makes it faster on real hardware, unless the pruning is structured (whole rows, channels or heads).
How it works
- Magnitude pruning: set the smallest-magnitude weights to zero, for example 50%, then fine-tune briefly to recover accuracy. Repeat in stages for higher sparsity.
- Unstructured pruning leaves the matrix the same shape with scattered zeros, which rarely runs faster on standard hardware.
- Structured pruning removes whole neurons, channels, attention heads or layers, so the model really shrinks and really speeds up.
- Distillation: run the teacher on training inputs and keep its full probability distribution (soft labels), not just the top answer.
- Soften both teacher and student with a temperature T > 1 (divide the logits by T), so small probabilities carry more signal.
- Train the student to match the teacher's softened probabilities (a KL loss, scaled by T²), usually mixed with the normal loss on true labels.
- For LLMs, distillation also means fine-tuning a small model on text generated by a large one.
Key points
- Soft labels carry "dark knowledge": which wrong answers are nearly right.
- Temperature softens distributions; T = 1 is the normal softmax.
- Structured pruning gives real speed-ups; unstructured mostly saves storage (unless hardware supports sparsity).
- Prune gradually and retrain in between: removing a little at a time lets the remaining weights make up for what was cut.
- Distillation, pruning and quantisation combine well.
Common mistakes
- Pruning heavily in one shot with no retraining.
- Expecting scattered zeros to make inference faster on a normal GPU.
- Distilling with T = 1 only, losing most of the soft-label information.
- Evaluating the student only on the teacher's strengths.
Not true: Pruning: remove most of the weights in one cut; no retraining is needed to keep the accuracy.
In fact: One heavy cut usually wrecks the accuracy. Prune a little at a time and retrain in between, so that the remaining weights can adjust.
Not true: Unstructured pruning: scattered zeros that make inference faster on an ordinary GPU.
In fact: A normal GPU still multiplies by the zeros. Scattered (unstructured) pruning mostly saves storage; removing whole neurons or channels gives real speed-ups.
Part of Unit 6: Systems and scale in the free ML Foundations course.