Partial derivatives, gradients and Jacobians
A partial derivative is the rate of change when you nudge one input and hold the others still. The gradient puts all of them into one vector, which points in the direction of steepest increase.
Why it matters
A loss depends on many weights at once, often millions. You need to know how it changes with each one, and which combined direction makes it fall fastest. The gradient packages exactly that, and gradient descent simply steps the other way. Jacobians extend the idea to functions with many outputs, like a whole network layer.
The idea
A partial derivative changes one input and holds the others still. The gradient collects all the partial derivatives into one vector, and it points uphill. The Jacobian is the same idea when the output is a vector too.
Standing on a hill. Step east: how steep is it? Step north: how steep? Those two numbers together are the gradient.
A hill has 2 directions. A network's loss has millions. You cannot picture it, but the maths is exactly the same.
How it works
- Take the bowl f(x, y) = x² + 2y² and stand at (1.5, 1), where f = 2.25 + 2 = 4.25.
- Partial derivative in x: treat y as a constant, so ∂f/∂x = 2x = 3. Stepping east, f rises 3 units per unit step.
- Partial derivative in y: treat x as a constant, so ∂f/∂y = 4y = 4.
- The gradient stacks them: ∇f = [3, 4]. It points straight uphill and is at right angles to the contour line through the point.
- Its length, √(3² + 4²) = 5, is the steepness in that best direction.
- Gradient descent steps the opposite way, −∇f = [−3, −4]. With learning rate 0.1 the next point is (1.5, 1) − 0.1·[3, 4] = (1.2, 0.6), where f = 2.16.
- For a function with several outputs, put each output's gradient in its own row: that matrix is the Jacobian. A layer y = Wx has Jacobian W.
Key points
- Partial derivative: vary one input and freeze the rest.
- Gradient ∇f = [∂f/∂x₁, …, ∂f/∂xₙ]: it points towards the steepest increase, and its length is the steepness.
- The gradient is at right angles to contour lines (lines of equal value).
- At a minimum, a maximum or a saddle point, the gradient is zero.
- Jacobian: one row per output, one column per input. For y = Wx it is W.
- The gradient of the loss has the same shape as the weights: one number per weight.
Common mistakes
- Forgetting to hold the other variables still: ∂/∂x of x²y is 2xy, not 2x.
- Stepping along +∇f when minimising. Descent goes against the gradient.
- Getting the Jacobian sideways. Fix the rule: rows are outputs, columns are inputs.
Not true: In a Jacobian, each row belongs to one input and each column to one output.
In fact: It is the other way round: one row for each output, one column for each input.
Not true: To make a function smaller, step in the direction of its gradient.
In fact: The gradient points uphill, the way the function grows fastest. To make it smaller, step against the gradient.
Part of Unit 1: Math foundations in the free ML Foundations course.