Metrics: precision, recall, F1, ROC-AUC, confusion matrix

Classification metrics count the four outcomes (true and false positives and negatives) and summarise them: precision, recall, F1, accuracy, and ROC-AUC across all thresholds.

Why it matters

Accuracy can lie. If 1% of transactions are fraud, a model that never flags anything is 99% accurate and completely useless. Different jobs care about different mistakes: a cancer screen must not miss cases (recall), a spam filter must not bin real email (precision). The right metric is what tells you whether a model does the job.

The idea

A confusion matrix counts true positives, false positives, false negatives and true negatives. Precision = TP/(TP + FP): of the items you flagged, how many were right? Recall = TP/(TP + FN): of the real ones, how many did you find? F1 balances the two. ROC-AUC measures how well scores rank positives above negatives.

Fishing with a net. Precision: how much of your catch is fish, not old boots? Recall: how many of the fish in the lake did you catch?

In fishing you can see your catch. In ML, the threshold you choose changes both numbers, so always say which threshold you used.

How it works

  1. Pick a threshold and turn scores into yes/no predictions.
  2. Count the four outcomes in a confusion matrix: TP (flagged, really positive), FP (flagged, really negative), FN (missed positive), TN (correctly left alone).
  3. Precision = TP / (TP + FP): of everything flagged, how much was right?
  4. Recall = TP / (TP + FN): of everything really positive, how much did we find?
  5. F1 = 2·P·R / (P + R): one number that is only high when both are high.
  6. Move the threshold: lower it and recall rises while precision usually falls; raise it and the reverse happens.
  7. ROC curve: plot recall (true positive rate) against false positive rate for every threshold. The area under it (AUC) is the chance a random positive scores above a random negative: 0.5 is guessing, 1.0 is perfect.

Key points

Common mistakes

Not true: A model with high precision must also have high recall.
In fact: They pull against each other. A model that says yes only when it is very sure has high precision but misses many positives, so its recall is low.

Not true: With 1% positives, an accuracy of 99% shows that a model is good.
In fact: A model that always answers "no" also scores 99%, and finds nothing. On imbalanced data, look at precision and recall.

Part of Unit 2: Classical ML, built by you in the free ML Foundations course.