Partial derivatives, gradients and Jacobians

A partial derivative is the rate of change when you nudge one input and hold the others still. The gradient puts all of them into one vector, which points in the direction of steepest increase.

Why it matters

A loss depends on many weights at once, often millions. You need to know how it changes with each one, and which combined direction makes it fall fastest. The gradient packages exactly that, and gradient descent simply steps the other way. Jacobians extend the idea to functions with many outputs, like a whole network layer.

The idea

A partial derivative changes one input and holds the others still. The gradient collects all the partial derivatives into one vector, and it points uphill. The Jacobian is the same idea when the output is a vector too.

Standing on a hill. Step east: how steep is it? Step north: how steep? Those two numbers together are the gradient.

A hill has 2 directions. A network's loss has millions. You cannot picture it, but the maths is exactly the same.

How it works

  1. Take the bowl f(x, y) = x² + 2y² and stand at (1.5, 1), where f = 2.25 + 2 = 4.25.
  2. Partial derivative in x: treat y as a constant, so ∂f/∂x = 2x = 3. Stepping east, f rises 3 units per unit step.
  3. Partial derivative in y: treat x as a constant, so ∂f/∂y = 4y = 4.
  4. The gradient stacks them: ∇f = [3, 4]. It points straight uphill and is at right angles to the contour line through the point.
  5. Its length, √(3² + 4²) = 5, is the steepness in that best direction.
  6. Gradient descent steps the opposite way, −∇f = [−3, −4]. With learning rate 0.1 the next point is (1.5, 1) − 0.1·[3, 4] = (1.2, 0.6), where f = 2.16.
  7. For a function with several outputs, put each output's gradient in its own row: that matrix is the Jacobian. A layer y = Wx has Jacobian W.

Key points

Common mistakes

Not true: In a Jacobian, each row belongs to one input and each column to one output.
In fact: It is the other way round: one row for each output, one column for each input.

Not true: To make a function smaller, step in the direction of its gradient.
In fact: The gradient points uphill, the way the function grows fastest. To make it smaller, step against the gradient.

Part of Unit 1: Math foundations in the free ML Foundations course.