Visual walkthrough — Monte Carlo simulation — law of large numbers basis

The characters, before any maths

Let me introduce every symbol so nothing arrives unannounced.

Step 1 — What one random draw looks like

WHAT. Before averaging anything, look at a single draw $X_i$. It's one dot landing somewhere, scattered around the true centre $\mu$.

WHY. You cannot understand an average of many things until you see one thing. The spread of that one dot is exactly what $\sigma$ measures — the typical distance from $\mu$.

PICTURE. Below, the yellow line is the true mean $\mu$. Blue dots are individual draws $X_i$; they land all over, sometimes far from $\mu$. The red band shows one $\sigma$ of typical spread.

$\underset{\text{one draw}}{\underbrace{X_i}}\approx \underset{\text{true centre}}{\underbrace{\mu }}\pm \underset{\text{typical wobble}}{\underbrace{\sigma }}$

Each single draw is noisy. A single poke tells you almost nothing. The magic is in the averaging.

Step 2 — The average sits exactly on $\mu$ (unbiased)

WHAT. We compute the expected value of the sample mean: on average, where is ${\bar{X}}_n$ centred?

WHY. An estimator is only trustworthy if it doesn't systematically aim too high or too low. We must confirm it points at the right target before worrying about spread.

$\mathbb{E}[{\bar{X}}_n]=\mathbb{E}\left[\underset{\text{constant}}{\underbrace{\frac{1}{n}}}\underset{\text{sum of draws}}{\underbrace{{\sum }_{i=1}^n{X}_i}}\right]=\frac{1}{n}{\sum }_{i=1}^n\underset{=\mu \text{ each}}{\underbrace{\mathbb{E}[X_i]}}=\frac{1}{n}\cdot n\mu =\mu .$

Term by term: the $\frac{1}{n}$ is a constant, so it slides out of the expectation. Linearity of expectation lets the $\mathbb{E}$ pass through the sum. Each of the $n$ pieces equals $\mu$, giving $n\mu$, and the $n$'s cancel.

PICTURE. The blue dots scatter, but their balance-point (the green marker, the average) sits right on the yellow $\mu$ line — no matter how few or many dots.

Step 3 — Averaging shrinks the spread by $1/n$ (in variance)

WHAT. We compute the variance of the sample mean — how far ${\bar{X}}_n$ itself typically strays from $\mu$.

WHY. Unbiased alone isn't enough: a dart-thrower centred on the bullseye but wild everywhere is useless. We need to prove the wildness of ${\bar{X}}_n$ falls as $n$ grows.

$\mathrm{Var}({\bar{X}}_n)=\mathrm{Var}(\underset{\text{pull out as }1/n^2}{\underbrace{\frac{1}{n}}}\sum X_i)=\frac{1}{n^2}\underset{\text{independence: no cross terms}}{\underbrace{\sum _{i=1}^n\mathrm{Var}(X_i)}}=\frac{1}{n^2}\cdot n{\sigma }^2=\frac{{\sigma }^2}{n}.$

Term by term: pulling the constant $\frac{1}{n}$ out of a variance squares it (rule: $\mathrm{Var}(aY)=a^2\mathrm{Var}(Y)$), so it becomes $\frac{1}{n^2}$. Independence is what kills all the cross-covariance terms — without it this step is false (see the degenerate case in Step 6). Each $\mathrm{Var}(X_i)={\sigma }^2$, and $n$ of them give $n{\sigma }^2$. The $n^2$ downstairs beats the single $n$ upstairs, leaving ${\sigma }^2/n$.

PICTURE. Three stacked clouds of the average dot: for $n=1$, $n=10$, $n=100$. As $n$ climbs, the cloud of averages collapses toward $\mu$.

Step 4 — From variance to a guarantee: Chebyshev's inequality

WHAT. We convert "small variance" into a hard statement about probability: how often can ${\bar{X}}_n$ land far from $\mu$?

WHY. Variance is an average of squared distances — helpful, but we want a promise like "the chance of being off by more than $\epsilon$ is tiny." Chebyshev's Inequality is exactly the bridge from variance to probability of a large miss, and it needs nothing but the variance.

$P\left(\underset{\text{miss by at least }\epsilon }{\underbrace{|{\bar{X}}_n-\mu |\ge \epsilon }}\right)\le \frac{\mathrm{Var}({\bar{X}}_n)}{{\epsilon }^2}=\frac{{\sigma }^2}{n{\epsilon }^2}.$

Term by term: $\epsilon$ (Greek "epsilon") is a tolerance we choose — "how close is close enough". The left side is the probability of missing by that much or more. Chebyshev caps it by variance divided by ${\epsilon }^2$. Substituting Step 3's ${\sigma }^2/n$ gives the final bound.

PICTURE. The bell of possible ${\bar{X}}_n$ values, with a tolerance band $\mu \pm \epsilon$ shaded green. Chebyshev bounds the red tail area outside the band by ${\sigma }^2/(n{\epsilon }^2)$.

Step 5 — Let $n\to \infty$: the tail is crushed to zero

WHAT. Hold $\epsilon$ fixed and let the sample size grow without bound.

WHY. This is the finish line. If the bound itself goes to $0$, the probability it caps is squeezed to $0$ too — that is convergence.

${\lim }_{n\to \infty }P(|{\bar{X}}_n-\mu |\ge \epsilon )\le {\lim }_{n\to \infty }\underset{\text{numerator fixed, denom }\to \infty }{\underbrace{\frac{{\sigma }^2}{n{\epsilon }^2}}}=0.$

Term by term: ${\sigma }^2$ and ${\epsilon }^2$ are frozen constants; only $n$ in the denominator grows. A fixed number over something exploding to infinity goes to $0$. Since a probability can't be negative, it's trapped at exactly $0$. This is the Weak Law of Large Numbers: ${\bar{X}}_n\stackrel{P}{\to }\mu$. $■$

PICTURE. The bound ${\sigma }^2/(n{\epsilon }^2)$ plotted against $n$ — a curve diving toward the axis. Overlaid: the actual bell of ${\bar{X}}_n$ narrowing to a spike on $\mu$.

Step 6 — The degenerate case: what breaks the guarantee

WHAT. Every step above secretly relied on two things: independence and a finite variance. Watch what happens when either fails.

WHY. The contract says cover every case. A reader who uses correlated samples or a heavy-tailed integrand and never saw this will be blindsided.

• Correlated draws. In Step 3, independence deleted the cross-terms. If draws are correlated, extra covariance terms survive, and the variance is larger than ${\sigma }^2/n$ — sometimes it never shrinks. The averaging barely helps.
• Infinite variance. In Step 4, Chebyshev needed a finite ${\sigma }^2$. For a heavy-tailed variable (e.g. a Cauchy-like $g$), ${\sigma }^2=\infty$ and the bound ${\sigma }^2/(n{\epsilon }^2)$ is useless. The sample mean can wander forever and never settle.

PICTURE. Two running-average traces vs $n$. Green: independent, finite variance — converges smoothly to $\mu$. Red: heavy-tailed — jumps and never settles.

The one-picture summary

Everything on one canvas: single noisy draws (Step 1) → their average lands on $\mu$ (Step 2) → averaging tightens the spread as ${\sigma }^2/n$ (Step 3) → Chebyshev caps the tail (Step 4) → $n\to \infty$ crushes it to zero (Step 5), except when independence or finite variance fails (Step 6).

Active Recall