Law of Large Numbers — weak and strong: Worked examples

This page is the drill room for the parent topic. We march through every kind of question the Law of Large Numbers (LLN) can ask, so that when the exam or a real problem hands you one, you've already seen its shape.

Before we start, plain-word reminders (so no symbol appears un-earned):

Definition The symbols we reuse everywhere

$X_{1}, X_{2}, \dots$ — a list of independent identical repeats of one random experiment (coin flip, dice roll, random draw). "Independent" = one outcome never influences another. "Identical" = every one is drawn from the same rulebook.
$μ = E [X_{i}]$ — the true mean, the long-run average value one draw would give. Think of it as the bullseye.
$\overset{ˉ}{X}_{n} = \frac{1}{n} \sum_{i = 1}^{n} X_{i}$ — the sample mean: add up your first $n$ results, divide by $n$ . This is the wandering dot that LLN says walks toward the bullseye.
$σ^{2} = Var (X_{i}) = E [(X_{i} - μ)^{2}]$ — the variance, the average squared distance of one draw from $μ$ . It measures how jumpy one draw is.
$ε$ (epsilon) — the error tolerance we choose: a small positive number, the "how close is close enough" radius around $μ$ . The event $∣ \overset{ˉ}{X}_{n} - μ ∣ \geq ε$ means "the sample mean missed the bullseye by at least $ε$ ."
$δ$ (delta) — a target ceiling on probability we choose: a small positive number, the largest failure chance we're willing to accept (e.g. $δ = 0.05$ means "at most a 5% chance of missing").

Our master tool is the WLLN bound proved in the parent note: $P (∣ \overset{ˉ}{X}_{n} - μ ∣ \geq ε) \leq \frac{σ ^{2}}{n ε ^{2}}$ Read it as: "the chance the sample mean misses the bullseye by at least $ε$ is capped by variance-over-(n times epsilon-squared)." This comes from Chebyshev's Inequality, which comes from Markov's Inequality.

Definition The three theorems by name (hypothesis → conclusion)

Different examples lean on different laws; here is each in one line so no name is used blind.

Chebyshev's route (the easy proof): hypothesis $σ^{2} < \infty$ (finite variance) → conclusion $P (∣ \overset{ˉ}{X}_{n} - μ ∣ \geq ε) \leq \frac{σ ^{2}}{n ε ^{2}} \to 0$ . This is what powers every bound on this page.
Khinchin's Weak Law: hypothesis only $E ∣ X_{i} ∣ < \infty$ (finite mean, variance may be infinite) → conclusion $\overset{ˉ}{X}_{n} \to μ$ in probability. It weakens the assumption of Chebyshev while keeping the Weak-Law conclusion.
Kolmogorov's Strong Law: hypothesis only $E ∣ X_{i} ∣ < \infty$ → conclusion $P (lim_{n} \overset{ˉ}{X}_{n} = μ) = 1$ , i.e. convergence almost surely (the stronger, whole-path statement). So Chebyshev buys a rate cheaply but needs variance; Khinchin and Kolmogorov need only a finite mean but work harder (via maximal inequalities and Borel–Cantelli).

The scenario matrix

Every LLN question falls into one of these case classes. Each example below is tagged with the cell(s) it covers.

#	Case class	What's special about it	Example
A	Bound a probability at fixed $n$	plug straight into $σ^{2} / (n ε^{2})$	Ex 1
B	Invert for $n$ (sample-size design)	solve the bound for $n$ , target $δ$	Ex 2
C	Non-binary variable (dice / general range)	compute $μ, σ^{2}$ first	Ex 3
D	Degenerate input $σ^{2} = 0$	a constant "random" variable	Ex 4
E	Limiting behaviour $n \to \infty$ , and $ε \to 0$	which knob wins	Ex 5
F	LLN fails — no finite mean	Cauchy / heavy tail	Ex 6
G	Real-world word problem	Monte Carlo estimate + error	Ex 7
H	Exam twist — Weak vs Strong on one path	distinguish the two modes	Ex 8
I	Fallacy trap — "streak is due"	Gambler's fallacy quantified	Ex 9

Worked examples

Ex 1 — Cell A · Bound a probability at fixed $n$

Ex 2 — Cell B · Invert for the sample size $n$

Ex 3 — Cell C · A non-binary variable (fair die)

Ex 4 — Cell D · Degenerate input, $σ^{2} = 0$

Ex 5 — Cell E · Limiting behaviour: which knob wins?

This is best seen. The band of "bad" outcomes must thin as $n$ grows, while shrinking $ε$ fights back.

Figure — Law of Large Numbers — weak and strong

Worked example

$n \to \infty$ vs $ε \to 0$ Coin ( $σ^{2} = \frac{1}{4}$ ). Compare the bound as (a) $n \to \infty$ with $ε$ fixed, (b) $ε \to 0$ with $n$ fixed.

Forecast: in case (b), does the guarantee survive?

Case (a): $\frac{1/4}{n ε ^{2}} n \to \infty 0.$ Why this step? $n$ sits in the denominator, so growing it drives the cap to $0$ — this is the Weak Law.
Case (b): with $n$ fixed, as $ε \to 0$ the raw expression $\frac{1/4}{n ε ^{2}} \to + \infty.$ Why this step? $ε^{2}$ shrinks in the denominator, blowing the expression up. But since a probability bound can't exceed $1$ , the usable bound is $min (1, \frac{1/4}{n ε ^{2}})$ , which is stuck at $1$ — vacuous, telling us nothing.
Lesson: you must send $n \to \infty$ first to shrink noise, and only then can you afford a tiny $ε$ . Look at the red curve ( $ε = 0.05$ ) in the figure: it's the highest one; halving $ε$ lifts the whole curve, but every curve still crawls to $0$ .

Answer: $n$ wins in the limit; $ε$ alone breaks the bound into vacuousness. Verify: at $n = 100, ε = 0.05$ : $\frac{0.25}{100 \cdot 0.0025} = 1.0$ — exactly the useless value $1$ ✓, confirming small $ε$ empties the guarantee. See Modes of Convergence for why "in probability" is the right lens here.

Ex 6 — Cell F · When LLN fails: the Cauchy trap

Worked example Averages that never settle

Draw $X_{i}$ from the standard Cauchy distribution (bell-ish but with tails so fat the mean is undefined). Does $\overset{ˉ}{X}_{n} \to$ anything?

Forecast: more draws should help... right?

Check the hypothesis: LLN requires $E ∣ X_{i} ∣ < \infty$ . For Cauchy, $\int ∣ x ∣ f (x) d x = \infty$ — no finite mean. Why this step? Every LLN theorem (Chebyshev, Khinchin, Kolmogorov) assumes at minimum a finite mean; without it there's no bullseye to converge to.
Structural fact: a remarkable property of Cauchy is that $\overset{ˉ}{X}_{n}$ has the same distribution as a single $X_{1}$ , for every $n$ . Why this step? Averaging Cauchys does not reduce spread at all — the $σ^{2} / n$ engine relies on finite variance, which is also infinite here.
Conclusion: $\overset{ˉ}{X}_{n}$ keeps jumping around forever; it converges to nothing.

Answer: LLN does not apply — no convergence. Verify: $Var (\overset{ˉ}{X}_{n}) = σ^{2} / n$ needs finite $σ^{2}$ ; Cauchy has $σ^{2} = \infty$ , so $\infty/ n = \infty$ — the engine is off ✓. This is the crucial "degenerate output" case: LLN is a theorem with hypotheses, not a law of nature.

Ex 7 — Cell G · Real-world word problem (Monte Carlo)

Worked example Estimating

$\int_{0}^{1} x^{2} d x$ by random sampling Draw $U_{i} \sim Uniform (0, 1)$ , average $g (U_{i}) = U_{i}^{2}$ . To what does $\hat{I}_{n} = \frac{1}{n} \sum U_{i}^{2}$ converge, and bound the error at $n = 10, 000, ε = 0.01$ ?

Forecast: guess the true integral first, then whether $10, 000$ samples pins it to $0.01$ .

Identify the target as an expectation: $E [U^{2}] = \int_{0}^{1} x^{2} d x = \frac{1}{3} .$ Why this step? An integral over $[0, 1]$ is the mean of $g (U)$ — this is why Monte Carlo Methods work: LLN turns integration into averaging.
By SLLN, $\hat{I}_{n} \to \frac{1}{3}$ almost surely. Why this step? $g (U_{i})$ are i.i.d. with finite mean, so the Strong Law guarantees the estimate lands on $\frac{1}{3}$ forever.
Error bound. Need $σ^{2} = Var (U^{2}) = E [U^{4}] - (E [U^{2}])^{2} = \frac{1}{5} - \frac{1}{9} = \frac{4}{45} \approx 0.0889.$ Why this step? Chebyshev needs the variance of one term; $E [U^{4}] = \int_{0}^{1} x^{4} d x = \frac{1}{5}$ .
Plug the bound: $\frac{σ ^{2}}{n ε ^{2}} = \frac{4/45}{10000 \cdot 0.0001} = \frac{0.0889}{1} \approx 0.0889.$

Answer: estimate $\to \frac{1}{3}$ ; at most $\approx 8.89%$ chance of missing by $0.01$ . Verify: $\frac{1}{5} - \frac{1}{9} = \frac{9 - 5}{45} = \frac{4}{45}$ ✓; the bound is a valid probability $\leq 1$ ✓. Want tighter? push $n$ up — the cap falls like $1/ n$ .

Ex 8 — Cell H · Exam twist: Weak vs Strong on one trajectory

The picture shows one random path of $\overset{ˉ}{X}_{n}$ so you can literally see the two claims.

Worked example Which law forbids re-escaping the funnel?

A student says: "By WLLN, once $\overset{ˉ}{X}_{n}$ is within $ε$ of $μ$ , it can never leave again." True or false, and which law addresses it?

Forecast: does "in probability" pin down a single path forever?

What WLLN actually claims: for each fixed large $n$ , $P (∣ \overset{ˉ}{X}_{n} - μ ∣ > ε)$ is small. It speaks about a snapshot, not the whole path. Why this step? Convergence in probability controls each $n$ separately — it permits rare later excursions.
What SLLN adds: $P (lim_{n} \overset{ˉ}{X}_{n} = μ) = 1$ — almost every entire path enters the funnel and stays. Why this step? Almost-sure convergence, built via the Borel–Cantelli Lemma summary above, forbids infinitely many escapes.
Verdict: the student's "can never leave" is a Strong Law statement, not Weak. WLLN alone does not forbid a later jump-out for a specific path.

Answer: the claim is a Strong-Law property, mis-attributed to the Weak Law — so as stated (crediting WLLN) it is false. Verify: in the figure, watch the yellow path briefly leave the $\pm ε$ green band at a large $n$ and re-enter — WLLN tolerates this, SLLN says such events stop happening eventually. Strong $\Rightarrow$ Weak, never the reverse.

Ex 9 — Cell I · Fallacy trap: is a tail "due"?

Worked example After 10 heads, quantify how "off" the average is

A fair coin lands heads 10 times in a row ( $n = 10$ , all heads). By how much is $\overset{ˉ}{X}_{10}$ off $μ$ , and does the next flip try to correct it?

Forecast: does the deviation shrink because of the streak, or despite it?

Current deviation: $\overset{ˉ}{X}_{10} = 1$ , so $\overset{ˉ}{X}_{10} - μ = 1 - 0.5 = 0.5.$ Why this step? Ten heads gives fraction $1$ ; subtract the true rate $0.5$ .
The sum of deviations: $\sum_{i = 1}^{10} (X_{i} - μ) = 10 \cdot 0.5 = 5.$ This raw surplus does not get pushed back down by the coin. Why this step? The $X_{i}$ are independent — the coin has no memory, so future flips carry no "correcting force." See Gambler's Fallacy.
How the average heals — by dilution: if the next $990$ flips are perfectly average (half heads), then $\overset{ˉ}{X}_{1000} = \frac{10 + 495}{1000} = \frac{505}{1000} = 0.505.$ Why this step? The surplus of $5$ is now shared among $1000$ flips: deviation drops from $0.5$ to $0.005$ without the streak ever being "cancelled."

Answer: the streak isn't reversed; it's outvoted as $n$ grows. Deviation went $0.5 \to 0.005$ . Verify: $\frac{505}{1000} - \frac{1}{2} = 0.005$ ✓, exactly $\frac{5}{1000}$ — the fixed surplus over a growing $n$ . This is the rigorous refutation of "tails is due."

Active recall

Recall Which cell does "solve for the smallest

$n$ " belong to, and what's the algebra? Cell B. Set $\frac{σ ^{2}}{n ε ^{2}} \leq δ$ (where $δ$ is your chosen probability ceiling) and invert: $n \geq \frac{σ ^{2}}{δ ε ^{2}}$ .

Recall Why does LLN fail for the Cauchy distribution?

No finite mean ( $E ∣ X ∣ = \infty$ ), so there is no $μ$ to converge to; $\overset{ˉ}{X}_{n}$ stays Cauchy-distributed for all $n$ .

Recall A streak of 10 heads: does the coin push back? How does the average recover?

No push-back (independence, the coin has no memory). Recovery is by dilution — the fixed surplus becomes a tiny fraction of a large $n$ , so the deviation shrinks even though the surplus itself never disappears.

Recall State Khinchin's and Kolmogorov's laws and how they differ from the Chebyshev proof.

Both need only $E ∣ X ∣ < \infty$ . Khinchin gives convergence in probability (Weak); Kolmogorov gives convergence almost surely (Strong). Chebyshev's proof needs the extra hypothesis $σ^{2} < \infty$ but rewards you with an explicit rate $σ^{2} / (n ε^{2})$ .

Deviation of

\overset{ˉ}{X}_{n}

for a constant

X_{i} = 7

exactly

0

for all

n

;

σ^{2} = 0

makes the bound vanish.

The knob that must go to infinity for WLLN

n

; shrinking

ε

alone makes the bound vacuous.

A probability bound that evaluates to 1 or more

is vacuous — it only says

P \leq 1

, which is always true.