Foundations — Central limit theorem

This page builds every symbol the parent note uses, starting from things a curious 12-year-old already knows: dice, coin flips, and dot-patterns. We add nothing until you can see it.

1. Randomness: the random variable $X$

Picture it. Roll one die. Before you look, the outcome is unknown — but it's one of $\{1,2,3,4,5,6\}$. That whole "machine that produces an unknown number" is $X$.

Why the topic needs it. Every piece of data in machine learning — a pixel value, a gradient, a click/no-click — is a number that came from some random process. We need one letter to talk about "a single draw" before we talk about many draws.

When we have many draws we number them: $X_1,X_2,\dots ,X_n$. The subscript is just a label ("draw number 1", "draw number 2"), and $n$ is how many draws we took.

2. The distribution: the shape of the randomness

Picture it. For a fair die every face has the same chance $1/6$, so its distribution is flat — six equal-height bars. We call this uniform because every value is equally likely.

Why the topic needs it. The whole magic of the CLT is that it doesn't care what this starting shape is — flat, lopsided, spiky, discrete or continuous — the averages still end up bell-shaped. To appreciate that, you must first be able to see the starting shape.

3. The mean $\mu$ and the symbol $\mathbb{E}[\cdot ]$

Picture it. Mark every possible value on a see-saw and put a weight on each equal to its probability. The point where the see-saw balances is $\mu$. For the die, the balance point sits exactly in the middle at $3.5$.

$\mu =\mathbb{E}[X]=\frac{1+2+3+4+5+6}{6}=3.5$

Why the topic needs it. $\mu$ is the true answer we are trying to estimate from data. The CLT is a statement about how close our average of samples gets to this $\mu$.

4. Spread: variance $\text{Var}(X)={\sigma }^2$ and standard deviation $\sigma$

Picture it. Two dartboards, both centred on $\mu$. One has darts clustered tight (small $\sigma$), one has darts scattered wide (large $\sigma$). Squaring the distances means a dart twice as far away counts four times as much, so outliers dominate.

Why the topic needs it. $\sigma$ measures how noisy a single measurement is. The CLT's headline result is that averaging shrinks this spread in a precise way — so we must be able to name the spread first.

5. Independence and "identically distributed" (i.i.d.)

Picture it. Imagine one factory printing dice, all identical (identically distributed), and you roll them in separate rooms so no roll can affect another (independent).

Why the topic needs it. Independence is what lets us add variances ($\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)$) — the key algebra step we use in the very next section and again in the CLT proof. "Identically distributed" is what lets us say each draw has the same $\mu$ and ${\sigma }^2$.

6. The sample mean ${\bar{X}}_n$

Reading the notation. The symbol ${\sum }_{i=1}^n{X}_i$ is just shorthand for "add up $X_1+X_2+\cdots +X_n$." The $\sum$ is a capital Greek S ("Sigma", for Sum); $i$ is the counter that walks from $1$ to $n$. Then $\frac{1}{n}$ divides by how many we added — that's an average.

Picture it. Every time you compute an average you get one new number. Do the whole experiment again and you get a slightly different average. So ${\bar{X}}_n$ is itself a random variable with its own distribution — and the CLT describes exactly that distribution.

Why the topic needs it. ${\bar{X}}_n$ is the star of the show. The CLT is a statement about what happens to ${\bar{X}}_n$ as $n$ grows.

7. Standard error: $\sigma /\sqrt{n}$

Picture it. As you average more numbers, the lucky-high and unlucky-low draws cancel out, so the average wobbles less. The $\sqrt{n}$ in the denominator says: to halve the wobble you need four times as many samples (since $\sqrt{4}=2$).

Why the topic needs it. This is the CLT's quantitative punch-line: averaging doesn't just make things bell-shaped, it makes the bell narrower at a rate of $1/\sqrt{n}$. This single fact powers the law of large numbers, Confidence Intervals, and Sample Size Calculation.

8. Standardizing: the $Z$-score

Picture it. Two bell curves of different widths sitting at different places. Standardizing slides each one so its centre lands on $0$, then stretches or squeezes it so its width becomes exactly $1$. Now they overlap perfectly — a fair comparison.

Why the topic needs it. The CLT compares the sample mean to a single reference curve, the standard normal. To make that comparison we first standardize ${\bar{X}}_n$: $Z_n=\frac{{\bar{X}}_n-\mu }{\sigma /\sqrt{n}}.$ Numerator: "how far is our average from the truth $\mu$." Denominator: the standard error. So $Z_n$ reads as "how many standard errors away are we?"

9. The normal distribution $\mathcal{N}(\mu ,{\sigma }^2)$ and $\Phi$

Picture it. A single humped curve, tallest at the centre, tailing off symmetrically on both sides. The full picture lives in Normal Distribution.

Why the topic needs it. Once the CLT tells us $Z_n$ is approximately standard normal, any probability question about the sample mean becomes an area under this one fixed curve — read off via $\Phi$. That is exactly what the parent's worked examples do.

10. The "converges in distribution" arrow $\stackrel{d}{\to }$

Picture it. A sequence of histograms, each for a bigger $n$, morphing frame by frame until it lies right on top of the bell curve.

Why the topic needs it. The CLT's core statement is $Z_n\stackrel{d}{\to }\mathcal{N}(0,1)$: the standardized sample mean's shape becomes the standard normal.

11. The fine print: finite variance $\mathbb{E}[X^2]<\infty$

Picture it. A "normal" distribution's tails shrink fast, so extreme values are rare enough that squared distances stay finite. A pathological heavy-tailed distribution keeps producing giant outliers whose squares dominate every average — the bell curve never forms.

Why the topic needs it. The parent's definition writes "$\text{Var}(X_i)={\sigma }^2<\infty$" — that little "$<\infty$" is this assumption. It's the one condition standing between "averaging tames randomness" and "averaging can't help you."

How these feed the theorem

Equipment checklist

Cover the right side and test yourself. If any answer is fuzzy, re-read that section before the main proof.

Ready? Continue to Central limit theorem for the full derivation, and see Law of Large Numbers for its close cousin.