Residual connections (ResNet)
A residual connection adds a block's input to its output. The block then only has to learn a change, and gradients get a direct path back through the network.
Why it matters
Before residuals, making a network deeper often made it worse, even on training data: the signal and gradients degraded through many layers, and optimisation got stuck. Residual connections fixed this so well that every modern deep architecture, including every transformer, uses them. Without them, today's 100-layer models would not train.
The idea
Instead of learning y = F(x), a residual block learns y = x + F(x). The block only has to learn the change. Gradients flow straight back through the '+ x' shortcut.
Editing a document with tracked changes, instead of rewriting it from scratch each time.
Tracked changes are optional. In a ResNet the shortcut is always there, and shapes must match before adding (a 1×1 convolution fixes a mismatch).
How it works
- A normal block computes y = F(x), for example conv → norm → ReLU → conv.
- A residual block computes y = x + F(x): the input skips around the block and is added back.
- If the best thing a block can do is nothing, it only has to learn F(x) = 0, which is easy. Adding a block can no longer make things worse.
- Backwards: dy/dx = 1 + F′(x). The 1 sends the gradient straight through, even when F′(x) is tiny.
- Chain many blocks and the gradient has a clean "highway" back to the earliest layers.
- When F changes the shape (more channels, smaller size), the shortcut uses a 1 × 1 convolution or a linear layer so the two can be added.
- In transformers every sub-layer is wrapped the same way: x = x + Attention(LayerNorm(x)), then x = x + MLP(LayerNorm(x)).
Key points
- y = x + F(x): learn the change, not the whole mapping.
- Gradient through a block = 1 + F′(x): the identity path keeps gradients alive.
- The shapes of x and F(x) must match before adding.
- Residual stream: in transformers, layers read from and write to one shared running vector.
- ResNet (2015) trained 152-layer networks this way and won ImageNet.
Common mistakes
- Shape mismatch on the shortcut when the channels change.
- Putting a ReLU after the addition in a way that blocks the identity path.
- Thinking residuals add parameters. The shortcut itself is free.
Not true: A residual connection adds extra weights that the network has to learn.
In fact: The shortcut is a plain addition with no weights. Only when the shapes differ does it need a small projection, such as a 1 × 1 convolution.
Not true: x and F(x) can be added even when their shapes differ.
In fact: Addition needs matching shapes. When F changes the number of channels or the size, the shortcut uses a projection, such as a 1 × 1 convolution, to match.
Part of Unit 3: Deep learning foundations in the free ML Foundations course.