Statistics: sampling, confidence intervals, hypothesis tests

Statistics measures how far a result from a sample can be trusted. A confidence interval gives a likely range for the true value, and a hypothesis test asks whether a difference could just be luck.

Why it matters

Every accuracy you report comes from a finite test set, so it is an estimate with noise. Model B beating model A by 1% might be real, or might vanish on another test set. Without confidence intervals and tests you will chase noise, ship worse models and over-claim results.

The idea

You usually see a sample, not everyone. A confidence interval gives a range that probably contains the true value. A hypothesis test asks: could this result be just luck?

Tasting one spoon of soup to judge the whole pot. Stirring first is random sampling.

Soup mixes easily. Real data often does not. If your sample is biased, for example only one type of user, more data will not fix it.

How it works

  1. Treat the test set as a random sample. Each example is right (1) or wrong (0), so accuracy is a sample mean.
  2. The standard error of a proportion is √(p(1 − p)/n) = √(0.85·0.15/200) ≈ 0.025.
  3. By the central limit theorem, averages are close to Gaussian, so a 95% interval is about ±1.96 standard errors: 85% ± 4.9%, roughly 80% to 90%.
  4. Meaning: if you repeated the whole experiment many times, about 95% of the intervals built this way would contain the true accuracy.
  5. Model B scores 86%, well inside A's interval, so this data can't tell the two apart.
  6. A hypothesis test makes this formal. Assume there is no real difference (the null hypothesis) and compute the p-value: the chance of a gap at least this big by luck alone. A small p-value, say below 0.05, is evidence of a real difference.
  7. To shrink the interval, get more data. The standard error falls like 1/√n, so 4 times more examples halves it.

Key points

Common mistakes

Not true: A p-value of 0.04 means there is a 96% chance that the new model really is better.
In fact: A p-value is the chance of a result at least this extreme if there were no real difference. It is not the probability that the effect is real.

Not true: If the best of 20 model variants passes p < 0.05, its improvement must be real.
In fact: With many variants, some look significant by luck alone: about 1 in 20 passes p < 0.05 with no real effect. Reporting only the best one hides that.

Go deeper: Why the standard error falls like 1/√n

  1. Take n independent samples X₁, …, Xₙ, each with mean μ and variance σ². Call their average m = (X₁ + … + Xₙ)/n.
  2. Two facts about variance: multiplying a variable by c multiplies its variance by c², and for independent variables the variance of a sum is the sum of the variances.
  3. So Var(X₁ + … + Xₙ) = nσ², and Var(m) = nσ² / n² = σ²/n.
  4. The standard error is the standard deviation of the estimate: √(σ²/n) = σ/√n.
  5. For accuracy, each test example scores 1 (right) with probability p or 0 (wrong). That variable has variance p(1 − p), so the standard error is √(p(1 − p)/n): the formula in the lesson.
  6. The catch is the word independent. If many test examples are near-copies of each other, the real uncertainty is larger than the formula says.

Var(the mean of n independent samples) = σ²/n, so the standard error is σ/√n. Four times the data halves it.

Part of Unit 1: Math foundations in the free ML Foundations course.