Residual connections (ResNet)

A residual connection adds a block's input to its output. The block then only has to learn a change, and gradients get a direct path back through the network.

Why it matters

Before residuals, making a network deeper often made it worse, even on training data: the signal and gradients degraded through many layers, and optimisation got stuck. Residual connections fixed this so well that every modern deep architecture, including every transformer, uses them. Without them, today's 100-layer models would not train.

The idea

Instead of learning y = F(x), a residual block learns y = x + F(x). The block only has to learn the change. Gradients flow straight back through the '+ x' shortcut.

Editing a document with tracked changes, instead of rewriting it from scratch each time.

Tracked changes are optional. In a ResNet the shortcut is always there, and shapes must match before adding (a 1×1 convolution fixes a mismatch).

How it works

  1. A normal block computes y = F(x), for example conv → norm → ReLU → conv.
  2. A residual block computes y = x + F(x): the input skips around the block and is added back.
  3. If the best thing a block can do is nothing, it only has to learn F(x) = 0, which is easy. Adding a block can no longer make things worse.
  4. Backwards: dy/dx = 1 + F′(x). The 1 sends the gradient straight through, even when F′(x) is tiny.
  5. Chain many blocks and the gradient has a clean "highway" back to the earliest layers.
  6. When F changes the shape (more channels, smaller size), the shortcut uses a 1 × 1 convolution or a linear layer so the two can be added.
  7. In transformers every sub-layer is wrapped the same way: x = x + Attention(LayerNorm(x)), then x = x + MLP(LayerNorm(x)).

Key points

Common mistakes

Not true: A residual connection adds extra weights that the network has to learn.
In fact: The shortcut is a plain addition with no weights. Only when the shapes differ does it need a small projection, such as a 1 × 1 convolution.

Not true: x and F(x) can be added even when their shapes differ.
In fact: Addition needs matching shapes. When F changes the number of channels or the size, the shortcut uses a projection, such as a 1 × 1 convolution, to match.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.