Debugging training: overfit one batch, read loss curves
Debugging training means checking, in a fixed order, that the model can learn at all — sensible starting loss, overfitting one small batch — before reading the loss curves of real runs.
Why it matters
Neural network bugs rarely crash. A shuffled label, a missing zero_grad or a wrong axis gives code that runs, prints a falling loss and quietly produces a worse model. Without a routine you can lose days tuning hyperparameters on top of a bug. A few cheap checks catch most problems in minutes.
The idea
First, make the model overfit one small batch: the loss should go close to 0. If it cannot, you have a bug. Then watch the training and validation loss curves together.
A mechanic tests the engine on a stand before driving it on the road.
An engine test is quick and clear. Training bugs can be silent: the code runs and the loss drops a bit, but the result is wrong. So also check the loss at step 0.
How it works
- Look at the data first: print a few examples with their labels, and check shapes, ranges and class balance.
- Check the loss at step 0. With K classes and random weights it should be about ln K: ln 10 ≈ 2.30 for 10 classes. A very different number means an initialisation or loss bug.
- Overfit one small batch (say 8 examples): turn regularisation off and train on just that batch. The loss should go close to 0. If it can't, there is a bug.
- Start from a simple baseline (a tiny model, a known learning rate) and change one thing at a time.
- Run a short learning-rate sweep (factors of 3 to 10) and pick the largest rate that still trains smoothly.
- Read the curves. Train down and validation up: overfitting. Both high and flat: underfitting, or the learning rate is too low. Spikes or NaN: the learning rate is too high, or there is a numerical bug.
- Watch gradient norms per layer: zeros mean a broken graph or dead units; huge values mean explosion.
Key points
- Initial loss ≈ ln(number of classes) for a classifier.
- Can't overfit one batch → bug, not a hyperparameter problem.
- Change one thing at a time, and keep a log of runs.
- Visualise predictions on real examples, not just the loss.
- Fix the random seed while debugging.
- Karpathy's "A Recipe for Training Neural Networks" is the classic checklist.
Common mistakes
- Tuning hyperparameters before confirming the model can overfit one batch.
- Trusting a falling loss without looking at predictions.
- Changing several things between runs, so you can't tell what helped.
- Comparing runs with different seeds and treating small differences as real.
Not true: A loss that keeps falling proves the model is learning the right thing.
In fact: A loss can fall while the predictions are useless, for example when the labels leak into the inputs. Always look at real predictions as well.
Not true: Hyperparameter tuning is the first thing to try when a model won't learn.
In fact: First check that the model can overfit one small batch. If it can't, there is a bug, and no hyperparameter will fix it.
Part of Unit 3: Deep learning foundations in the free ML Foundations course.