Metrics: precision, recall, F1, ROC-AUC, confusion matrix
Classification metrics count the four outcomes (true and false positives and negatives) and summarise them: precision, recall, F1, accuracy, and ROC-AUC across all thresholds.
Why it matters
Accuracy can lie. If 1% of transactions are fraud, a model that never flags anything is 99% accurate and completely useless. Different jobs care about different mistakes: a cancer screen must not miss cases (recall), a spam filter must not bin real email (precision). The right metric is what tells you whether a model does the job.
The idea
A confusion matrix counts true positives, false positives, false negatives and true negatives. Precision = TP/(TP + FP): of the items you flagged, how many were right? Recall = TP/(TP + FN): of the real ones, how many did you find? F1 balances the two. ROC-AUC measures how well scores rank positives above negatives.
Fishing with a net. Precision: how much of your catch is fish, not old boots? Recall: how many of the fish in the lake did you catch?
In fishing you can see your catch. In ML, the threshold you choose changes both numbers, so always say which threshold you used.
How it works
- Pick a threshold and turn scores into yes/no predictions.
- Count the four outcomes in a confusion matrix: TP (flagged, really positive), FP (flagged, really negative), FN (missed positive), TN (correctly left alone).
- Precision = TP / (TP + FP): of everything flagged, how much was right?
- Recall = TP / (TP + FN): of everything really positive, how much did we find?
- F1 = 2·P·R / (P + R): one number that is only high when both are high.
- Move the threshold: lower it and recall rises while precision usually falls; raise it and the reverse happens.
- ROC curve: plot recall (true positive rate) against false positive rate for every threshold. The area under it (AUC) is the chance a random positive scores above a random negative: 0.5 is guessing, 1.0 is perfect.
Key points
- Accuracy = (TP + TN) / everything. Misleading when classes are imbalanced.
- Precision: "when it says yes, is it right?" Recall: "does it find them all?"
- There is always a trade-off between them, controlled by the threshold.
- ROC-AUC doesn't depend on the threshold. With very rare positives, the precision–recall curve is more telling.
- Always report the threshold you used, and compare against a dumb baseline.
- Choose the metric from the cost of each mistake, before you train.
Common mistakes
- Reporting accuracy on imbalanced data: with 1% positives, always answering "no" scores 99%.
- Quoting precision without recall, or the other way round.
- Choosing the threshold on the test set.
- Mixing up FP and FN. Write out the 2 × 2 table every time.
Not true: A model with high precision must also have high recall.
In fact: They pull against each other. A model that says yes only when it is very sure has high precision but misses many positives, so its recall is low.
Not true: With 1% positives, an accuracy of 99% shows that a model is good.
In fact: A model that always answers "no" also scores 99%, and finds nothing. On imbalanced data, look at precision and recall.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.