Linear regression: normal equation and gradient descent

Linear regression predicts a number as a weighted sum of the features. It chooses the weights that make the mean squared error as small as possible.

Why it matters

Many questions ask for a number: a price, a temperature, a delivery time. Linear regression is the simplest model that learns how each feature pushes the answer up or down. It trains in a fraction of a second, its weights are easy to read, and it is the baseline every fancier model must beat.

Its two training routes, a formula and gradient descent, are the same two routes used everywhere else in ML.

The idea

Predict with a weighted sum, ŷ = Xw + b, and choose the weights that make the squared errors smallest. There are two ways: a direct formula (the normal equation), or gradient descent step by step.

Placing a ruler across dots on a page so it is as close as possible to all of them.

A ruler is straight. If the true pattern curves, linear regression cannot follow it, unless you add curved features such as x².

How it works

  1. Put the examples in a matrix X (one row each) and add a column of 1s, so the bias b becomes just another weight.
  2. The prediction for every row at once is ŷ = Xw.
  3. The loss is the mean squared error: the average of (ŷ − y)². Squaring makes big misses count much more than small ones.
  4. Route 1, the normal equation: set the gradient to zero and solve XᵀX w = Xᵀy in one step. For x = [1, 2, 3, 4, 5] and y = [2, 4, 5, 4, 6] this gives ŷ = 1.8 + 0.8x.
  5. Route 2, gradient descent: the gradient is (2/n)·Xᵀ(Xw − y). Step against it, again and again, and w walks to the same answer.
  6. The normal equation is exact and fast for a few thousand features. Gradient descent scales to huge data and is how every neural network trains.
  7. Read the weights: 0.8 means "one more unit of x adds 0.8 to the prediction, holding the other features fixed".

Key points

Common mistakes

Not true: A large weight in a linear regression shows that its feature causes the outcome.
In fact: A weight only says how the prediction moves when that feature changes and the others stay still. It shows an association, not a cause.

Not true: Without a bias term: the fitted line can still cross the y-axis where the data needs it to.
In fact: Without the bias, the line is forced through the origin. The bias (a column of ones) lets the line sit at any height.

Part of Unit 2: Classical ML, built by you in the free ML Foundations course.