Logistic regression and the sigmoid

Logistic regression answers yes/no questions. It computes a score, squashes it into a probability with the sigmoid, and is trained with cross-entropy loss.

Why it matters

Many questions have a yes/no answer: spam or not, fraud or not, pass or fail. A straight line gives numbers below 0 and above 1, which can't be probabilities. Logistic regression keeps the simple weighted sum but squashes it into a probability, so it can say "83% likely". It is the standard first model for classification, and the single neuron at the heart of every neural network.

The idea

For yes/no questions. Compute a score z = Xw + b, then squash it into a probability with the sigmoid, σ(z) = 1 / (1 + e⁻ᶻ). Train with cross-entropy loss.

A dimmer switch. The score turns the dial, and the sigmoid keeps the light between 0 (off) and 1 (fully on).

The decision boundary is still a straight line (a flat plane). It cannot separate data shaped like circles unless you add new features.

How it works

  1. Compute a score for each example, exactly as in linear regression: z = Xw + b.
  2. Squash it: p = σ(z) = 1 / (1 + e⁻ᶻ). A score of 0 gives 0.5, z = 2 gives 0.88, and z = −2 gives 0.12.
  3. Measure the error with binary cross-entropy: −[y log p + (1 − y) log(1 − p)]. Confident mistakes cost the most.
  4. The gradient comes out beautifully simple: Xᵀ(p − y)/n for the weights and the mean of (p − y) for the bias.
  5. Train with gradient descent. There is no closed-form answer, but the loss is convex, so gradient descent finds the best weights.
  6. To decide, pick a threshold (usually 0.5). p > 0.5 exactly when z > 0, so the decision boundary is the line (or plane) where Xw + b = 0.

Key points

Common mistakes

Not true: A threshold of 0.5 is the right cut-off even when one kind of mistake costs far more.
In fact: The threshold should follow the costs. When missing a positive is much worse than a false alarm, a lower threshold is better.

Not true: Loss for logistic regression: squared error works as well as cross-entropy.
In fact: With a sigmoid, squared error makes the loss non-convex and learning slow. Binary cross-entropy is convex, and its gradient is the simple Xᵀ(p − y)/n.

Part of Unit 2: Classical ML, built by you in the free ML Foundations course.