Conditional probability and Bayes’ rule
Conditional probability is the chance of A once you know that B happened. Bayes' rule flips it round: from the chance of B given A, it gives the chance of A given B, using how common A was to begin with.
Why it matters
People and models constantly update beliefs from evidence: is this email spam given these words, is this patient sick given a positive test?
The probability you know (how often the test is positive for sick people) is usually not the one you want (how likely you are sick given a positive test). Bayes' rule converts one into the other, and ignoring the starting rate is one of the most common reasoning errors there is.
The idea
P(A | B) is the probability of A when you already know B happened. Bayes' rule flips it around: P(A | B) = P(B | A) · P(A) / P(B).
A smoke alarm beeps. Is there a fire? The alarm is good at detecting fires, but fires are rare, and burnt toast is common.
From experience, you roughly know how rare fires are. In real problems the starting probability P(A), the prior, is often unknown, and the answer depends on it.
How it works
- Start with counts, not formulas. Imagine 1,000 people; 1% are sick, so 10 are sick and 990 are healthy.
- The test catches 99% of sick people: about 9.9 of the 10 test positive.
- It also raises a false alarm for 5% of healthy people: 0.05 × 990 = 49.5 test positive.
- Of all 59.4 positives, only 9.9 are sick. So P(sick | positive) = 9.9 / 59.4 ≈ 0.17.
- That is Bayes' rule: P(sick | +) = P(+ | sick)·P(sick) / P(+), where P(+) adds up both ways of testing positive.
- The starting rate (the prior) matters enormously. If 5% were sick, the same test would give P(sick | +) ≈ 0.51.
Key points
- Conditional probability: P(A | B) = P(A and B) / P(B), the share of the B cases in which A happens too.
- Bayes: posterior = likelihood × prior / evidence.
- P(A | B) ≠ P(B | A). The test is 99% sensitive, yet a positive result means only a 17% chance of being sick.
- Natural frequencies (counting 1,000 people) make Bayes problems far easier to get right.
- Independence means P(A | B) = P(A): knowing B tells you nothing about A.
- Naive Bayes classifiers (Unit 2) and Bayesian reasoning about uncertainty are built on this rule.
Common mistakes
- Confusing P(positive | sick) with P(sick | positive).
- Ignoring the base rate (the prior). Rare things stay fairly unlikely even after quite strong evidence.
- Forgetting that P(B) must count every way B can happen.
Not true: Once the evidence is strong, how rare something was to begin with no longer matters.
In fact: The starting rate still matters a great deal: a rare disease stays fairly unlikely even after a positive result from a good test.
Not true: P(positive | sick) and P(sick | positive) are the same number.
In fact: They answer different questions and can be far apart: most sick people can test positive while most positive tests come from healthy people. Bayes' rule turns one into the other.
Part of Unit 1: Math foundations in the free ML Foundations course.