Naive Bayes

Naive Bayes classifies with Bayes' rule, assuming the features are independent once you know the class, so each feature's evidence can simply be multiplied in.

Why it matters

Text has thousands of word features but often little labelled data. Naive Bayes trains in a single counting pass, works well with small datasets, and makes a hard-to-beat baseline for spam filtering and document classification. It is also the clearest real use of Bayes' rule in ML.

The idea

Naive Bayes uses Bayes' rule to classify, with a 'naive' shortcut: it assumes features are independent once you know the class. Then you just multiply probabilities.

A spam filter reading one word at a time. 'free' makes spam more likely, 'meeting' makes it less likely. It adds up the evidence from each word separately.

Words are not really independent: 'New' and 'York' appear together. The probabilities come out wrong, but the ranking of classes is often still right.

How it works

  1. From the training data, count the prior: P(spam) = the share of emails that are spam.
  2. For each word, count how often it appears in spam and in ham (not spam): P(word | spam) and P(word | ham).
  3. For a new email, start each class at its prior and multiply in P(word | class) for every word in it.
  4. The bigger product wins. Divide each by their sum to get probabilities.
  5. In practice, add logs instead of multiplying, because a product of hundreds of small numbers rounds to 0.
  6. Add-one (Laplace) smoothing: add 1 to every count, so a word never seen in spam doesn't force P(spam) to exactly 0.

Key points

Common mistakes

Not true: Naive Bayes gives well-calibrated probabilities: when it says 0.99, it is right 99% of the time.
In fact: Treating the features as independent counts the same evidence several times, which pushes its probabilities towards 0 and 1. Trust its ranking of the classes, not the numbers.

Not true: Naive Bayes only works when the features really are independent.
In fact: The assumption is false for real text, where words depend on each other, yet the ranking of the classes is often still right. That makes it a useful baseline.

Part of Unit 2: Classical ML, built by you in the free ML Foundations course.