Naive Bayes
Naive Bayes classifies with Bayes' rule, assuming the features are independent once you know the class, so each feature's evidence can simply be multiplied in.
Why it matters
Text has thousands of word features but often little labelled data. Naive Bayes trains in a single counting pass, works well with small datasets, and makes a hard-to-beat baseline for spam filtering and document classification. It is also the clearest real use of Bayes' rule in ML.
The idea
Naive Bayes uses Bayes' rule to classify, with a 'naive' shortcut: it assumes features are independent once you know the class. Then you just multiply probabilities.
A spam filter reading one word at a time. 'free' makes spam more likely, 'meeting' makes it less likely. It adds up the evidence from each word separately.
Words are not really independent: 'New' and 'York' appear together. The probabilities come out wrong, but the ranking of classes is often still right.
How it works
- From the training data, count the prior: P(spam) = the share of emails that are spam.
- For each word, count how often it appears in spam and in ham (not spam): P(word | spam) and P(word | ham).
- For a new email, start each class at its prior and multiply in P(word | class) for every word in it.
- The bigger product wins. Divide each by their sum to get probabilities.
- In practice, add logs instead of multiplying, because a product of hundreds of small numbers rounds to 0.
- Add-one (Laplace) smoothing: add 1 to every count, so a word never seen in spam doesn't force P(spam) to exactly 0.
Key points
- Naive Bayes scores each class as prior × Π P(featureᵢ | class): how common the class is, times the probability of each feature given that class.
- The "naive" assumption is false for real text, yet the ranking of classes is often still right.
- Training is just counting: very fast, works on small data.
- Always add log-probabilities instead of multiplying probabilities, and use smoothing so that one unseen word can't make the whole product zero.
- Words that are equally common in both classes add no evidence.
- The probabilities it outputs are badly calibrated (too extreme); trust the ranking more than the numbers.
Common mistakes
- Multiplying raw probabilities, which underflow to 0 on long documents.
- No smoothing: one unseen word sends a class to probability 0.
- Treating its probabilities as well calibrated.
Not true: Naive Bayes gives well-calibrated probabilities: when it says 0.99, it is right 99% of the time.
In fact: Treating the features as independent counts the same evidence several times, which pushes its probabilities towards 0 and 1. Trust its ranking of the classes, not the numbers.
Not true: Naive Bayes only works when the features really are independent.
In fact: The assumption is false for real text, where words depend on each other, yet the ranking of the classes is often still right. That makes it a useful baseline.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.