Central limit theorem

Definition

WHAT does each piece mean?

• i.i.d.: Each sample is drawn independently from the same distribution
• $\mu$: The true population mean we're estimating
• $\sigma^2$: The true population variance (spread of individual data points)
• $\sigma/\sqrt{n}$: Standard error of the mean—how much sample means vary around $\mu$

Derivation from First Principles

GOAL: Prove that standardized sample means approach $\mathcal{N}(0, 1)$.

Step 1: Why Standardize?

We want to compare distributions of different scales. Standardization transforms any random variable to zero mean and unit variance: $Z = \frac{X - \mathbb{E}[X]}{\sqrt{\text{Var}(X)}}$

WHY this transformation? It makes the comparison "fair"—all variables now live on the same scale.

For the sample mean:

• $\mathbb{E}[\bar{X}_n] = \mathbb{E}\left[\frac{1}{n}\sum_{i=1}^{n}X_i\right] = \frac{1}{n}\sum_{i=1}^{n}\mathbb{E}[X_i] = \frac{1}{n}(n\mu) = \mu$ ✓
• $\text{Var}(\bar{X}_n) = \text{Var}\left(\frac{1}{n}\sum_{i=1}^{n}X_i\right)$

WHY can we pull out $1/n^2$? Because $\text{Var}(cX) = c^2\text{Var}(X)$: $= \frac{1}{n^2}\text{Var}\left(\sum_{i=1}^{n}X_i\right)$

WHY can we sum the variances? Independence: $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$ when $X \perp Y$: $= \frac{1}{n^2}\sum_{i=1}^{n}\text{Var}(X_i) = \frac{1}{n^2}(n\sigma^2) = \frac{\sigma^2}{n}$

Therefore: $\bar{X}_n$ has mean $\mu$ and variance $\sigma^2/n$.

Step 2: Moment Generating Functions (The Proof Engine)

The moment generating function (MGF) of a random variable $Y$ is: $M_Y(t) = \mathbb{E}[e^{tY}]$

WHY use MGFs? Key theorem: If $M_{Y_n}(t) \to M_Y(t)$ for all $t$, then $Y_n \xrightarrow{d} Y$. MGFs uniquely characterize distributions.

For standard normal: $M_{\mathcal{N}(0,1)}(t) = e^{t^2/2}$

Step 3: MGF of Standardized Sample Mean

Define standardized variables: $Y_i = \frac{X_i - \mu}{\sigma}$

Then $\mathbb{E}[Y_i] = 0$, $\text{Var}(Y_i) = 1$, and: $Z_n = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} = \frac{\sum_{i=1}^{n}(X_i - \mu)}{n\cdot\sigma/\sqrt{n}} = \frac{1}{\sqrt{n}}\sum_{i=1}^{n}Y_i$

WHY this form? We've reduced the problem to standardized variables.

The MGF is: $M_{Z_n}(t) = \mathbb{E}\left[\exp\left(\frac{t}{\sqrt{n}}\sum_{i=1}^{n}Y_i\right)\right]$

WHY can we split the product? Independence + exponential property $e^{a+b} = e^a \cdot e^b$: $= \mathbb{E}\left[\prod_{i=1}^{n}\exp\left(\frac{t}{\sqrt{n}}Y_i\right)\right] = \prod_{i=1}^{n}\mathbb{E}\left[\exp\left(\frac{t}{\sqrt{n}}Y_i\right)\right]$

Since $Y_i$ are identically distributed: $= \left[\mathbb{E}\left[e^{tY_1/\sqrt{n}}\right]\right]^n = \left[M_{Y_1}\left(\frac{t}{\sqrt{n}}\right)\right]^n$

Step 4: Taylor Expansion

WHY Taylor series? As $n \to \infty$, $t/\sqrt{n} \to 0$, so we can approximate the MGF.

For small $s$: $M_{Y_1}(s) = \mathbb{E}[e^{sY_1}] = \mathbb{E}\left[1 + sY_1 + \frac{s^2Y_1^2}{2} + \frac{s^3Y_1^3}{6} + \cdots\right]$

$= 1 + s\mathbb{E}[Y_1] + \frac{s^2}{2}\mathbb{E}[Y_1^2] + \frac{s^3}{6}\mathbb{E}[Y_1^3] + O(s^4)$

Since $\mathbb{E}[Y_1] = 0$ and $\mathbb{E}[Y_1^2] = 1$: $M_{Y_1}(s) = 1 + \frac{s^2}{2} + O(s^3)$

Substitute $s = t/\sqrt{n}$: $M_{Y_1}\left(\frac{t}{\sqrt{n}}\right) = 1 + \frac{t^2}{2n} + O\left(\frac{1}{n^{3/2}}\right)$

Step 5: Taking the Limit

$M_{Z_n}(t) = \left[1 + \frac{t^2}{2n} + O\left(\frac{1}{n^{3/2}}\right)\right]^n$

WHY does this approach $e^{t^2/2}$? Recall the limit definition of $e^x$: $\lim_{n\to\infty}\left(1 + \frac{x}{n}\right)^n = e^x$

Here $x = t^2/2$: $\lim_{n\to\infty}M_{Z_n}(t) = \lim_{n\to\infty}\left(1 + \frac{t^2/2}{n}\right)^n = e^{t^2/2}$

This is the MGF of $\mathcal{N}(0, 1)$. Therefore $Z_n \xrightarrow{d} \mathcal{N}(0, 1)$. ∎

Examples

Common Mistakes

Active Recall Questions

#flashcards/ai-ml

What is the Central Limit Theorem in one sentence? :: The CLT states that the distribution of sample means approaches a normal distribution as sample size increases, regardless of the population's distribution (assuming finite variance).

How does the standard error of the mean change with sample size? :: Standard error = σ/√n, so it decreases proportionally to the square root of n. To halve the error, you need 4× the sample size.

Connections

• Law of Large Numbers: Related but distinct—LN says sample mean converges to population mean; CLT describes the distribution of that convergence
• Normal Distribution: CLT explains why Gaussians appear everywhere in nature—they're the limit of averaged processes
• Confidence Intervals: CLT provides theoretical foundation for constructing CIs around sample means
• Hypothesis Testing: t-tests and z-tests rely on CLT for validity with large samples
• Stochastic Gradient Descent: CLT explains why noisy mini-batch gradients lead to convergence
• Bootstrap Methods: Non-parametric alternative when CLT assumptions (independence, finite variance) fail
• Sample Size Calculation: Uses CLT to determine how much data needed for desired precision
• Monte Carlo Methods: Simulation accuracy improves via CLT—averaging many random samples
• Maximum Likelihood Estimation: Asymptotic normality of MLEs is a consequence of CLT
• Bias-Variance Tradeoff: CLT explains why variance of estimators decreases as O(1/n)

Concept Map

Hinglish (regional understanding)