Random variables, distributions, expectation and variance

A random variable is a number decided by chance; its distribution says how likely each value is, its expectation is the long-run average, and its variance measures how spread out it is.

Why it matters

Data is noisy and models are never certain. Classifiers output probabilities, starting weights are drawn at random, mini-batches are random samples, and generated text is sampled. To reason about any of it — how big an error might be, how much results vary from run to run — you need the language of distributions, averages and spread.

The idea

A random variable is a number decided by chance. Its distribution says how likely each value is. The expectation is the long-run average. The variance measures how spread out the values are.

Rolling a die. Each face has probability 1/6. The expectation is 3.5: you never roll 3.5, but it is the average over many rolls.

A die has a few possible values. Many things in ML are continuous, like height, so we use a density curve instead of a list.

How it works

  1. List the outcomes and their probabilities: a fair die gives 1 to 6, each with probability 1/6. The probabilities always add up to 1.
  2. Expectation: weight each value by its probability and add. E[X] = (1 + 2 + … + 6)/6 = 3.5.
  3. Variance: the expected squared distance from the mean. Var(X) = E[X²] − E[X]² = 91/6 − 12.25 ≈ 2.92.
  4. The standard deviation is its square root, about 1.71: a typical distance from the average, in the same units as X.
  5. Simulate to check. The average of many rolls approaches 3.5 (the law of large numbers), and the histogram approaches the flat bars of the distribution.
  6. Continuous values, like height, use a density curve instead of a list, and probabilities are areas under the curve. The Gaussian N(μ, σ²) is the bell curve with mean μ and spread σ.

Key points

Common mistakes

Not true: Expectation E[X]: the most likely value of X.
In fact: It is the long-run average, which need not be a possible value at all: a fair die has expectation 3.5 and never rolls 3.5.

Not true: Density: like a probability, it lies between 0 and 1.
In fact: A density can be bigger than 1. Only areas under the curve are probabilities, and the whole area is 1.

Part of Unit 1: Math foundations in the free ML Foundations course.