Conditional probability and Bayes’ rule

Conditional probability is the chance of A once you know that B happened. Bayes' rule flips it round: from the chance of B given A, it gives the chance of A given B, using how common A was to begin with.

Why it matters

People and models constantly update beliefs from evidence: is this email spam given these words, is this patient sick given a positive test?

The probability you know (how often the test is positive for sick people) is usually not the one you want (how likely you are sick given a positive test). Bayes' rule converts one into the other, and ignoring the starting rate is one of the most common reasoning errors there is.

The idea

P(A | B) is the probability of A when you already know B happened. Bayes' rule flips it around: P(A | B) = P(B | A) · P(A) / P(B).

A smoke alarm beeps. Is there a fire? The alarm is good at detecting fires, but fires are rare, and burnt toast is common.

From experience, you roughly know how rare fires are. In real problems the starting probability P(A), the prior, is often unknown, and the answer depends on it.

How it works

  1. Start with counts, not formulas. Imagine 1,000 people; 1% are sick, so 10 are sick and 990 are healthy.
  2. The test catches 99% of sick people: about 9.9 of the 10 test positive.
  3. It also raises a false alarm for 5% of healthy people: 0.05 × 990 = 49.5 test positive.
  4. Of all 59.4 positives, only 9.9 are sick. So P(sick | positive) = 9.9 / 59.4 ≈ 0.17.
  5. That is Bayes' rule: P(sick | +) = P(+ | sick)·P(sick) / P(+), where P(+) adds up both ways of testing positive.
  6. The starting rate (the prior) matters enormously. If 5% were sick, the same test would give P(sick | +) ≈ 0.51.

Key points

Common mistakes

Not true: Once the evidence is strong, how rare something was to begin with no longer matters.
In fact: The starting rate still matters a great deal: a rare disease stays fairly unlikely even after a positive result from a good test.

Not true: P(positive | sick) and P(sick | positive) are the same number.
In fact: They answer different questions and can be far apart: most sick people can test positive while most positive tests come from healthy people. Bayes' rule turns one into the other.

Part of Unit 1: Math foundations in the free ML Foundations course.