Linear regression: normal equation and gradient descent
Linear regression predicts a number as a weighted sum of the features. It chooses the weights that make the mean squared error as small as possible.
Why it matters
Many questions ask for a number: a price, a temperature, a delivery time. Linear regression is the simplest model that learns how each feature pushes the answer up or down. It trains in a fraction of a second, its weights are easy to read, and it is the baseline every fancier model must beat.
Its two training routes, a formula and gradient descent, are the same two routes used everywhere else in ML.
The idea
Predict with a weighted sum, ŷ = Xw + b, and choose the weights that make the squared errors smallest. There are two ways: a direct formula (the normal equation), or gradient descent step by step.
Placing a ruler across dots on a page so it is as close as possible to all of them.
A ruler is straight. If the true pattern curves, linear regression cannot follow it, unless you add curved features such as x².
How it works
- Put the examples in a matrix X (one row each) and add a column of 1s, so the bias b becomes just another weight.
- The prediction for every row at once is ŷ = Xw.
- The loss is the mean squared error: the average of (ŷ − y)². Squaring makes big misses count much more than small ones.
- Route 1, the normal equation: set the gradient to zero and solve XᵀX w = Xᵀy in one step. For x = [1, 2, 3, 4, 5] and y = [2, 4, 5, 4, 6] this gives ŷ = 1.8 + 0.8x.
- Route 2, gradient descent: the gradient is (2/n)·Xᵀ(Xw − y). Step against it, again and again, and w walks to the same answer.
- The normal equation is exact and fast for a few thousand features. Gradient descent scales to huge data and is how every neural network trains.
- Read the weights: 0.8 means "one more unit of x adds 0.8 to the prediction, holding the other features fixed".
Key points
- Model ŷ = Xw + b; loss = mean squared error.
- Normal equation: w = (XᵀX)⁻¹Xᵀy. In code, solve the system (np.linalg.solve or lstsq); never form the inverse.
- The gradient of the mean squared error with respect to the weights is (2/n)·Xᵀ(Xw − y).
- Minimising squared error is maximum likelihood when the noise is Gaussian.
- Curves need curved features: add x², x³ or interactions, and the model is still "linear" in its weights.
- Check the residuals (y − ŷ): a pattern in them means the model is missing something.
Common mistakes
- Using it when the relationship is strongly curved, without adding features.
- Reading a weight as a cause. It only says how the prediction moves, with the other features held still.
- Duplicated or near-duplicate features: XᵀX becomes nearly singular and the weights become huge.
- Forgetting the bias column, which forces the line through the origin.
Not true: A large weight in a linear regression shows that its feature causes the outcome.
In fact: A weight only says how the prediction moves when that feature changes and the others stay still. It shows an association, not a cause.
Not true: Without a bias term: the fitted line can still cross the y-axis where the data needs it to.
In fact: Without the bias, the line is forced through the origin. The bias (a column of ones) lets the line sit at any height.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.