Foundations — Law of Large Numbers — weak and strong

This is the ground-floor page for Law of Large Numbers — weak and strong. The parent note throws a lot of notation at you very fast: $X_i$, $\mu$, ${\sigma }^2$, $\mathbb{E}$, ${\bar{X}}_n$, $\to$ in two different flavours, $\mathbb{P}(\dots )$, $\epsilon$, and the summation sign $\sum$. Below we build each one from nothing, in an order where every symbol is defined before it is used.

0. The picture we keep coming back to

Before any symbols, look at what the whole topic is about: a running average that starts wild and calms down.

The jagged cyan line is the average-so-far after each new trial. Notice it swings hugely at the left (few trials) and hugs the amber line at the right (many trials). The amber horizontal line is the number the average is heading for. Naming these two things precisely is the entire job of this page.

1. A random variable — $X$

Plain words. Flip a coin. The outcome is "heads" or "tails" — words, not numbers. To do arithmetic (averaging!) we need numbers, so we agree on a translation: heads $\to 1$, tails $\to 0$. That translation-rule is the random variable $X$.

The picture. Think of a machine: outcome goes in the slot, a number pops out.

Why the topic needs it. The Law of Large Numbers averages numbers. You cannot average "heads, tails, heads". You can average $1,0,1$. So step one is always: turn outcomes into numbers with a random variable.

2. Many copies, and the subscript $X_i$

Plain words. The subscript is just a label — trial number 1, 2, 3, … Nothing more mysterious than seat numbers in a cinema.

The picture. A row of identical machines from Section 1, each spitting out its own number, labelled left to right.

Why the topic needs it. LLN is about repetition. We need names for the 1st, 2nd, … results so we can add them up.

Two words attached to these copies: i.i.d.

Why the topic needs it. "Identical" guarantees there's one true mean $\mu$ to converge to. "Independent" is what makes the random wiggles cancel instead of piling up — it's literally the engine of the whole law (you'll see it kill the cross-terms in the variance calculation).

3. Probability — $\mathbb{P}(\dots )$

Plain words. Read $\mathbb{P}(A)$ as "the probability of $A$". Whatever sits inside the brackets is an event — a description of outcomes you care about, like "$X=1$" (heads) or "$|{\bar{X}}_n-\mu |>0.1$" (the average is far off).

The picture. A bar from $0$ to $1$; the event's likelihood is a mark somewhere along it.

Why the topic needs it. The Weak Law is a statement about a probability shrinking to zero. Without this symbol you cannot even write it.

4. Expectation / the mean — $\mathbb{E}[X]$ and $\mu$

This is the target the whole topic points at, so we go slowly.

Why the sum, and why this tool? We want the "centre of gravity" of the possible values. Weighting each value by how often it happens, then adding, is exactly the balance-point of a see-saw — heavier (more likely) values pull the balance toward themselves. That balance point is what a huge pile of averages should sit on. No other combination has that balancing property, which is why expectation, not (say) the middle value, is the right tool.

The picture. Values as weights on a ruler; $\mu$ is the point where the ruler balances.

Why the topic needs it. $\mu$ is the number the sample average converges to. It's the amber line in Figure 1.

5. The summation sign — $\sum$

Plain words. It's a compact "add all of these". ${\sum }_{i=1}^3{X}_i=X_1+X_2+X_3$.

The picture. A hopper that swallows the whole row of machine-outputs and pours out one total.

Why the topic needs it. An average is a total divided by a count. We need a clean way to write "the total of the first $n$ results".

6. The sample mean — ${\bar{X}}_n$

Now we can assemble the star of the show from parts we already own.

Plain words. Add up the first $n$ results, divide by how many there were. The bar on top means "average of", the subscript $n$ means "using $n$ trials so far".

The picture. This is exactly the jagged cyan line in Figure 1. Each new trial adds one term to the top and bumps $n$ up by one, giving a new dot on the curve.

Why the topic needs it. The entire Law of Large Numbers is the claim "${\bar{X}}_n$ approaches $\mu$". This symbol is the left-hand side of everything.

7. Spread — variance ${\sigma }^2$

Why square, and why this tool? We want one number for "typical wobble size". If we averaged the raw distance $X-\mu$, positives and negatives would cancel and always give $0$ — useless. Squaring removes the sign so nothing cancels, and it punishes big misses more. That is why variance, not average signed distance, is the tool that measures spread.

The picture. The width of the cloud of dots around $\mu$: a tight cloud has small ${\sigma }^2$, a wide scattered cloud has large ${\sigma }^2$.

Why the topic needs it. The speed at which ${\bar{X}}_n$ settles is governed by variance: the parent's engine $\mathrm{Var}({\bar{X}}_n)={\sigma }^2/n$ shrinks because we divide ${\sigma }^2$ by $n$. Big spread = slower settling.

8. The tolerance $\epsilon$ and "far off"

The vertical bars. $|z|$ is the absolute value: it strips the sign, giving the distance of $z$ from $0$. So $|{\bar{X}}_n-\mu |$ is just "how far apart the average and the true mean are", regardless of which is bigger.

The picture. A funnel of half-width $\epsilon$ drawn around the amber line. "Being off by more than $\epsilon$" means the cyan dot pokes outside the funnel.

Why the topic needs it. Convergence means "eventually inside every funnel, no matter how thin". $\epsilon$ is how we say "no matter how thin".

9. Two arrows: convergence — $\to$

The parent uses convergence in two different senses. Both mean "gets close and stays close", but they measure closeness differently. Details live in Modes of Convergence; here is the minimum you need.

• In probability (Weak Law): $\mathbb{P}(|{\bar{X}}_n-\mu |>\epsilon )\to 0$. For a fixed huge $n$, the chance of poking outside the funnel is tiny.
• Almost surely (Strong Law): $\mathbb{P}({\lim }_{n\to \infty }{\bar{X}}_n=\mu )=1$. Look at a whole infinite path; with certainty it eventually enters the funnel forever.

Why the topic needs both. The Weak and Strong Laws differ only in which arrow they use. Getting these two straight is the point of the parent note.

Prerequisite map

How to read it. Everything on the left is raw vocabulary; each arrow means "is needed to build". They all funnel into the two convergence statements, which are the Weak and Strong Laws. The variance branch feeds the speed of convergence, powering the Chebyshev's Inequality proof.

Where these lead next

• $\mathbb{E}$, $\mathbb{P}$, variance $\Rightarrow$ Markov's Inequality and Chebyshev's Inequality (the tools that prove the Weak Law).
• The two arrows $\Rightarrow$ Modes of Convergence (weak vs strong made rigorous).
• Fluctuation shape around $\mu$ $\Rightarrow$ Central Limit Theorem.
• Almost-sure machinery $\Rightarrow$ Borel–Cantelli Lemma.
• Averaging to estimate integrals $\Rightarrow$ Monte Carlo Methods.
• The classic misuse $\Rightarrow$ Gambler's Fallacy.

Equipment checklist

Self-test: cover the right side, answer, then reveal.