Support vector machines: margin and hinge loss
A support vector machine separates two classes with the line (or plane) that leaves the widest possible margin to the nearest points, trained with the hinge loss.
Why it matters
Many lines can separate two classes, but the one that hugs some points will misclassify slightly different new points. The SVM picks the boundary with the most room to spare, which tends to generalise well. Its maths — margins, hinge loss, the dual problem and kernels — also introduced ideas used all over ML.
The idea
An SVM finds the separating line (or plane) with the widest gap, the margin, to the nearest points of each class. Those nearest points are the support vectors. It uses the hinge loss, max(0, 1 − y·score).
Building a road between two villages. You want the widest road possible that does not touch any house.
Real data is messy: some houses sit on the wrong side. Soft-margin SVMs allow a few violations, at a cost set by C.
How it works
- Use labels −1 and +1, and a score s = w·x + b. The boundary is where s = 0.
- The margin lines are where s = +1 and s = −1. Their distance apart is 2/‖w‖, so a smaller ‖w‖ means a wider street.
- Hard margin: make every point satisfy y·s ≥ 1 (on the right side, outside the street) while keeping ‖w‖ as small as possible.
- Only the points lying on the margin lines decide the answer: the support vectors. Move any other point slightly and nothing changes.
- Soft margin: real data overlaps, so allow violations and pay for them with the hinge loss, max(0, 1 − y·s).
- Minimise (λ/2)‖w‖² + average hinge loss with subgradient descent. C = 1/λ: a big C punishes violations hard (narrow street); a small C allows more (wide street).
- Kernels replace dot products with a similarity function, giving curved boundaries without building the curved features by hand.
Key points
- The margin is 2/‖w‖ wide, so a smaller ‖w‖ means a wider margin.
- Hinge loss max(0, 1 − y·s): 0 for confident correct points, growing linearly for violations.
- Support vectors are the only points that matter for the final boundary.
- C trades margin width against training mistakes.
- The problem is convex, so there is one best answer.
- Scale features first: margins are measured in distances.
Common mistakes
- Using 0/1 labels with the ±1 formulas.
- Forgetting to scale features. The margin is a distance, so a feature with large numbers dominates it.
- Reading SVM scores as probabilities. They are distances, not probabilities.
Not true: An SVM's score for a point is the probability that the point belongs to the positive class.
In fact: The score is a signed distance from the boundary, not a probability. Turning it into a probability needs a separate calibration step.
Not true: The SVM formulas work unchanged with labels of 0 and 1.
In fact: The margin and the hinge loss use y·s, which needs labels of −1 and +1. With a label of 0, y·s is always 0 and the formulas break.
Go deeper: Why the margin is exactly 2/‖w‖
- The two margin lines are w·x + b = +1 and w·x + b = −1. The vector w points straight across the street: it is at 90° to both lines.
- Take any point x₁ on the −1 line and walk along w until you hit the +1 line. Walking a distance d in the direction of w takes you to x₂ = x₁ + d·w/‖w‖.
- x₂ is on the +1 line, so w·x₂ + b = 1. Substitute x₂: (w·x₁ + b) + d·(w·w)/‖w‖ = 1.
- Since w·x₁ + b = −1 and w·w = ‖w‖², this reads −1 + d·‖w‖ = 1. So d = 2/‖w‖.
- To make the street as wide as possible, make ‖w‖ as small as possible while every point stays on its own side: y·(w·x + b) ≥ 1. Minimising ‖w‖²/2 gives the same answer and is easier to differentiate.
The margin lines are 2/‖w‖ apart, so "maximise the margin" becomes "minimise ‖w‖² subject to y·(w·x + b) ≥ 1": a convex problem with one best answer.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.