Foundations — Monte Carlo simulation — law of large numbers basis

This page assumes nothing. Every squiggle in the parent note — $\mathbb{E}$, $\mu$, ${\sigma }^2$, ${\bar{X}}_n$, $\sum$, $1_A$, $\stackrel{P}{\to }$ — is built here, from a picture, before you are allowed to use it. Read top to bottom; each block earns the next.

0. What is a "random variable" $X$?

Plain words. A random variable is just a number you don't know yet because it depends on the outcome of some random experiment. Roll a die → the number on top is a random variable. Throw a dart at a square → whether it lands in a circle (yes/no, written 1/0) is a random variable.

The picture. Think of a machine with a lever. Pull the lever and out drops a number. You can't predict which — but you can predict the pattern if you pull many times.

Why the topic needs it. Everything Monte Carlo estimates ($\pi$, an integral) gets rewritten as "the typical value that drops out of some machine." We call the letter for that value $X$ (or $g(X)$ when we feed $X$ through a function $g$ first).

1. The Uniform machine and the indicator $1$

Two special machines appear constantly in the parent note.

The picture. The indicator is a light switch tied to a rule. Land inside the region → light ON (1). Land outside → light OFF (0). Its average is exactly the fraction of time the light is on — which is a probability. That single fact is the engine behind estimating $\pi$.

2. The summation sign $\sum$ and the sample mean ${\bar{X}}_n$

Why the topic needs it. ${\bar{X}}_n$ is the Monte Carlo estimator. Everything else on the page is about how close ${\bar{X}}_n$ gets to the truth and how fast.

3. Expectation $\mathbb{E}[X]$ and the mean $\mu$ — the "true average"

Now the crucial distinction that the whole subject turns on.

The picture. Imagine the histogram of a million pulls. It has a balance point — the spot where a ruler under it would perfectly balance. That balance point is $\mu =\mathbb{E}[X]$.

4. Variance ${\sigma }^2$ — how much the machine wobbles

Why squared? We square the distance $X-\mu$ so that being below the mean (negative gap) and above it (positive gap) both count as spread instead of cancelling. Deep dive: Variance and Covariance.

The picture. Small ${\sigma }^2$ = a tall thin pile hugging $\mu$ (consistent machine). Large ${\sigma }^2$ = a wide flat pile (erratic machine). This "width" is exactly what shrinks as you average more samples — the secret of why Monte Carlo works.

5. "i.i.d." — the independence you keep hearing about

The picture. $n$ identical machines, each pulled once by a different blindfolded person in a different room. Same design (identical), no communication (independent).

Why the topic needs it. Identical gives every $X_i$ the same $\mu$ so their average targets $\mu$. Independent is what lets variance simply add (§4). Break independence — use correlated samples — and the clean ${\sigma }^2/n$ formula collapses. See Chebyshev's Inequality for the bound that turns this spread into a guarantee.

6. The convergence arrow $\stackrel{P}{\to }$ and the limit

The picture. Draw a narrow band of half-width $\epsilon$ around the horizontal line $\mu$. As $n$ climbs, the wobbling path of ${\bar{X}}_n$ dives into that band and (almost) never leaves. A stronger version of this promise is the Strong Law of Large Numbers.

7. $\theta$, $g(X)$ and ${\hat{\theta }}_n$ — dressing up any target as an average

8. The standard error ${\sigma }_g/\sqrt{n}$ — why "root-n"

Once variance of the sample mean is ${\sigma }_g^2/n$ (§4), its standard deviation is the square root: $\text{SE}=\sqrt{\frac{{\sigma }_g^2}{n}}=\frac{{\sigma }_g}{\sqrt{n}}.$

Why the  ​? Variance is in squared units (§4); to get a spread in the same units as the answer you take the square root, which drags the $n$ under a root too. That is the origin of the famous "$1/\sqrt{n}$": to shrink error $10\times$ you need $n$ to grow $100\times$. The precise shape of the error's bell curve — and hence confidence intervals — comes from the Central Limit Theorem.

Prerequisite map

Equipment checklist

Cover the right side and see if you can state each before revealing.