Debugging training: overfit one batch, read loss curves

Debugging training means checking, in a fixed order, that the model can learn at all — sensible starting loss, overfitting one small batch — before reading the loss curves of real runs.

Why it matters

Neural network bugs rarely crash. A shuffled label, a missing zero_grad or a wrong axis gives code that runs, prints a falling loss and quietly produces a worse model. Without a routine you can lose days tuning hyperparameters on top of a bug. A few cheap checks catch most problems in minutes.

The idea

First, make the model overfit one small batch: the loss should go close to 0. If it cannot, you have a bug. Then watch the training and validation loss curves together.

A mechanic tests the engine on a stand before driving it on the road.

An engine test is quick and clear. Training bugs can be silent: the code runs and the loss drops a bit, but the result is wrong. So also check the loss at step 0.

How it works

  1. Look at the data first: print a few examples with their labels, and check shapes, ranges and class balance.
  2. Check the loss at step 0. With K classes and random weights it should be about ln K: ln 10 ≈ 2.30 for 10 classes. A very different number means an initialisation or loss bug.
  3. Overfit one small batch (say 8 examples): turn regularisation off and train on just that batch. The loss should go close to 0. If it can't, there is a bug.
  4. Start from a simple baseline (a tiny model, a known learning rate) and change one thing at a time.
  5. Run a short learning-rate sweep (factors of 3 to 10) and pick the largest rate that still trains smoothly.
  6. Read the curves. Train down and validation up: overfitting. Both high and flat: underfitting, or the learning rate is too low. Spikes or NaN: the learning rate is too high, or there is a numerical bug.
  7. Watch gradient norms per layer: zeros mean a broken graph or dead units; huge values mean explosion.

Key points

Common mistakes

Not true: A loss that keeps falling proves the model is learning the right thing.
In fact: A loss can fall while the predictions are useless, for example when the labels leak into the inputs. Always look at real predictions as well.

Not true: Hyperparameter tuning is the first thing to try when a model won't learn.
In fact: First check that the model can overfit one small batch. If it can't, there is a bug, and no hyperparameter will fix it.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.