Statistics: sampling, confidence intervals, hypothesis tests
Statistics measures how far a result from a sample can be trusted. A confidence interval gives a likely range for the true value, and a hypothesis test asks whether a difference could just be luck.
Why it matters
Every accuracy you report comes from a finite test set, so it is an estimate with noise. Model B beating model A by 1% might be real, or might vanish on another test set. Without confidence intervals and tests you will chase noise, ship worse models and over-claim results.
The idea
You usually see a sample, not everyone. A confidence interval gives a range that probably contains the true value. A hypothesis test asks: could this result be just luck?
Tasting one spoon of soup to judge the whole pot. Stirring first is random sampling.
Soup mixes easily. Real data often does not. If your sample is biased, for example only one type of user, more data will not fix it.
How it works
- Treat the test set as a random sample. Each example is right (1) or wrong (0), so accuracy is a sample mean.
- The standard error of a proportion is √(p(1 − p)/n) = √(0.85·0.15/200) ≈ 0.025.
- By the central limit theorem, averages are close to Gaussian, so a 95% interval is about ±1.96 standard errors: 85% ± 4.9%, roughly 80% to 90%.
- Meaning: if you repeated the whole experiment many times, about 95% of the intervals built this way would contain the true accuracy.
- Model B scores 86%, well inside A's interval, so this data can't tell the two apart.
- A hypothesis test makes this formal. Assume there is no real difference (the null hypothesis) and compute the p-value: the chance of a gap at least this big by luck alone. A small p-value, say below 0.05, is evidence of a real difference.
- To shrink the interval, get more data. The standard error falls like 1/√n, so 4 times more examples halves it.
Key points
- The standard error of a mean is σ/√n; of a proportion, such as an accuracy, it is √(p(1 − p)/n).
- A 95% confidence interval is about the estimate ± 1.96 standard errors.
- Central limit theorem: averages of many independent samples are roughly Gaussian, whatever the original distribution.
- A p-value is P(a result at least this extreme | no real effect). It is not the probability that the effect is real.
- 4 times more data halves the interval width.
- The bootstrap (resample your test set with replacement many times) gives an interval for any metric without formulas.
Common mistakes
- Reporting one accuracy number with no interval.
- Reading p = 0.04 as "a 96% chance the model is better".
- Trying many variants and reporting only the best: some will look significant by luck alone.
- Assuming a bigger sample fixes a biased one. It doesn't.
Not true: A p-value of 0.04 means there is a 96% chance that the new model really is better.
In fact: A p-value is the chance of a result at least this extreme if there were no real difference. It is not the probability that the effect is real.
Not true: If the best of 20 model variants passes p < 0.05, its improvement must be real.
In fact: With many variants, some look significant by luck alone: about 1 in 20 passes p < 0.05 with no real effect. Reporting only the best one hides that.
Go deeper: Why the standard error falls like 1/√n
- Take n independent samples X₁, …, Xₙ, each with mean μ and variance σ². Call their average m = (X₁ + … + Xₙ)/n.
- Two facts about variance: multiplying a variable by c multiplies its variance by c², and for independent variables the variance of a sum is the sum of the variances.
- So Var(X₁ + … + Xₙ) = nσ², and Var(m) = nσ² / n² = σ²/n.
- The standard error is the standard deviation of the estimate: √(σ²/n) = σ/√n.
- For accuracy, each test example scores 1 (right) with probability p or 0 (wrong). That variable has variance p(1 − p), so the standard error is √(p(1 − p)/n): the formula in the lesson.
- The catch is the word independent. If many test examples are near-copies of each other, the real uncertainty is larger than the formula says.
Var(the mean of n independent samples) = σ²/n, so the standard error is σ/√n. Four times the data halves it.
Part of Unit 1: Math foundations in the free ML Foundations course.