Backpropagation by hand

Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule from the output backwards, reusing each intermediate result.

Why it matters

Gradient descent needs one gradient per weight, and a network can have billions of weights. Nudging each weight separately to measure its effect would need billions of forward passes. Backprop gets every gradient at the cost of about one extra pass, which is what makes training deep networks possible at all.

The idea

Backprop computes the gradient of the loss for every weight, using the chain rule from the output backwards. Each step passes back: (gradient from above) × (its own local derivative).

A factory makes a faulty product. You trace back station by station, asking how much each one contributed to the fault.

In a factory, blame often goes to one station. In backprop, blame is shared, and it is added up across every path.

How it works

  1. Forward pass: compute and remember every intermediate value. For L = (w·x − y)² with w = 1, x = 2 and y = 5: ŷ = 2, e = ŷ − y = −3, L = 9.
  2. Start at the end: dL/dL = 1.
  3. Each operation passes back (the gradient arriving from above) × (its own local derivative). L = e², whose local derivative is 2e = −6, so dL/de = −6.
  4. e = ŷ − y has local derivative 1 with respect to ŷ, so dL/dŷ = −6.
  5. ŷ = w·x has local derivative x = 2 with respect to w, so dL/dw = −6 × 2 = −12.
  6. When one value feeds several later operations, the gradients coming back along each path are added together.
  7. Gradient descent then uses it: w ← w − η·(−12), so w increases, which lowers the loss.

Key points

Common mistakes

Not true: Backpropagation needs one separate forward pass for every weight in the network.
In fact: One forward pass and one backward pass give the gradient for every weight at once. That is what makes training large networks possible.

Not true: When a value is used in two places, only the last gradient that reaches it counts.
In fact: Gradients that reach the same value along different paths are added together. Overwriting instead of adding is a classic backprop bug.

Go deeper: Backprop in matrix form: where the transposes come from

  1. Set-up: a batch X (n rows, d columns) goes through weights W (d × k) to give Y = XW (n × k). Backprop has already produced G = dL/dY, a grid the same shape as Y. We want dL/dW and dL/dX.
  2. Write one entry of the product: Yᵢⱼ = Σₘ Xᵢₘ·Wₘⱼ. The weight Wₘⱼ appears only in column j of Y, once for each row i, and its local derivative there is ∂Yᵢⱼ/∂Wₘⱼ = Xᵢₘ.
  3. The chain rule adds up every path from Wₘⱼ to the loss: ∂L/∂Wₘⱼ = Σᵢ Gᵢⱼ·Xᵢₘ.
  4. That sum is entry (m, j) of the product Xᵀ·G, because row m of Xᵀ is column m of X. So dL/dW = Xᵀ·G.
  5. Do the same for an input. Xᵢₘ appears only in row i of Y, with local derivative Wₘⱼ. So ∂L/∂Xᵢₘ = Σⱼ Gᵢⱼ·Wₘⱼ, which is entry (i, m) of G·Wᵀ. So dL/dX = G·Wᵀ.
  6. Check the shapes. Xᵀ·G is (d × n)(n × k) = d × k, the shape of W. G·Wᵀ is (n × k)(k × d) = n × d, the shape of X. A gradient always has the shape of the thing it belongs to, and that rule alone tells you where the transposes go.

For Y = XW: dL/dW = Xᵀ·(dL/dY) and dL/dX = (dL/dY)·Wᵀ. Every layer of a network is this rule, plus the slope of its activation.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.