Maximum likelihood estimation
Maximum likelihood estimation chooses the parameter values that make the data you actually observed as probable as possible.
Why it matters
A model has unknown parameters: the bias of a coin, the mean of a Gaussian, the weights of a network. You need a principled rule for choosing them from data. Maximum likelihood is that rule, and it explains where the standard loss functions come from: squared error and cross-entropy are both maximum likelihood in disguise.
The idea
Maximum likelihood: choose the model parameters that make the data you actually saw most probable.
A detective picks the story that best explains all the clues.
A detective also uses common sense. MLE uses only the data, so with little data it can be overconfident: 3 heads in 3 flips gives 'this coin always lands heads'.
How it works
- Write the probability of the data as a function of the parameter: the likelihood. For 7 heads and 3 tails, L(p) = p⁷(1 − p)³.
- Try some values: L(0.5) ≈ 0.00098, L(0.7) ≈ 0.00222, L(0.9) ≈ 0.00048. The data is most probable at p = 0.7.
- Take the log, which turns products into sums and keeps the same peak: log L = 7 log p + 3 log(1 − p).
- Set the derivative to zero: 7/p − 3/(1 − p) = 0, so p = 7/10. The estimate is just the fraction of heads.
- In ML you minimise the negative log-likelihood instead of maximising the likelihood. That is your loss.
- For a Gaussian with fixed spread, maximising the likelihood is the same as minimising squared error, so the best mean is the sample mean. For classification, it is the same as minimising cross-entropy.
Key points
- Likelihood L(θ) = P(data | θ), seen as a function of the parameters θ.
- Maximise log L instead: products become sums, and tiny numbers don't underflow to 0.
- The negative log-likelihood is the loss you minimise.
- Squared error = Gaussian maximum likelihood. Cross-entropy = maximum likelihood for categories.
- For a coin, the estimate is the fraction of heads; for a Gaussian mean, it is the sample average.
- MAP adds a prior and maximises likelihood × prior. Weight decay is MAP with a Gaussian prior on the weights.
Common mistakes
- Reading the likelihood as "the probability that the parameter is right". It is the probability of the data, given the parameter.
- Multiplying thousands of small probabilities directly. They underflow to 0; add their logs instead.
- Trusting maximum likelihood on tiny data: 3 heads out of 3 flips gives p = 1.
Not true: Maximum likelihood on 3 coin flips: a trustworthy estimate of p.
In fact: Three heads out of three gives p = 1: a coin that could never land tails. On tiny data, add a prior or collect more.
Not true: Likelihood L(θ) = the probability that θ is the correct value.
In fact: It is the probability of the data, given that parameter value. It says how well the value explains what was seen.
Part of Unit 1: Math foundations in the free ML Foundations course.