Logistic regression and the sigmoid
Logistic regression answers yes/no questions. It computes a score, squashes it into a probability with the sigmoid, and is trained with cross-entropy loss.
Why it matters
Many questions have a yes/no answer: spam or not, fraud or not, pass or fail. A straight line gives numbers below 0 and above 1, which can't be probabilities. Logistic regression keeps the simple weighted sum but squashes it into a probability, so it can say "83% likely". It is the standard first model for classification, and the single neuron at the heart of every neural network.
The idea
For yes/no questions. Compute a score z = Xw + b, then squash it into a probability with the sigmoid, σ(z) = 1 / (1 + e⁻ᶻ). Train with cross-entropy loss.
A dimmer switch. The score turns the dial, and the sigmoid keeps the light between 0 (off) and 1 (fully on).
The decision boundary is still a straight line (a flat plane). It cannot separate data shaped like circles unless you add new features.
How it works
- Compute a score for each example, exactly as in linear regression: z = Xw + b.
- Squash it: p = σ(z) = 1 / (1 + e⁻ᶻ). A score of 0 gives 0.5, z = 2 gives 0.88, and z = −2 gives 0.12.
- Measure the error with binary cross-entropy: −[y log p + (1 − y) log(1 − p)]. Confident mistakes cost the most.
- The gradient comes out beautifully simple: Xᵀ(p − y)/n for the weights and the mean of (p − y) for the bias.
- Train with gradient descent. There is no closed-form answer, but the loss is convex, so gradient descent finds the best weights.
- To decide, pick a threshold (usually 0.5). p > 0.5 exactly when z > 0, so the decision boundary is the line (or plane) where Xw + b = 0.
Key points
- σ(z) = 1/(1 + e⁻ᶻ) maps any number to between 0 and 1; σ(0) = 0.5.
- Train with binary cross-entropy, not squared error.
- Gradient: Xᵀ(p − y)/n. Remember it — it appears again in neural networks.
- The boundary is linear. Curved boundaries need extra features.
- The weights are log-odds: one more unit of a feature adds w to log(p/(1 − p)).
- More than two classes: replace the sigmoid with softmax (Unit 3).
Common mistakes
- Training with squared error. The loss becomes non-convex and learning is slow.
- Reading the output as certain truth. The probabilities are only as good as the model and the data.
- Perfectly separable data with no regularisation: the weights grow forever. Add L2.
- Always using 0.5 as the threshold, even when one kind of mistake costs much more.
Not true: A threshold of 0.5 is the right cut-off even when one kind of mistake costs far more.
In fact: The threshold should follow the costs. When missing a positive is much worse than a false alarm, a lower threshold is better.
Not true: Loss for logistic regression: squared error works as well as cross-entropy.
In fact: With a sigmoid, squared error makes the loss non-convex and learning slow. Binary cross-entropy is convex, and its gradient is the simple Xᵀ(p − y)/n.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.