Law of Large Numbers — weak and strong

WHAT are we actually claiming?

Let $X_1, X_2, \dots$ be i.i.d. (independent, identically distributed) random variables with finite mean $\mu = \mathbb{E}[X_i]$ . Define the sample mean:

$\bar X_n = \frac{1}{n}\sum_{i=1}^n X_i$

There are two flavours of " $\bar X_n \to \mu$ ", because there are two ways a sequence of random variables can converge.

HOW to derive the Weak Law (from scratch)

We build it from two elementary inequalities. Derive these too.

Step 1 — Markov's inequality

For a non-negative random variable $Y\ge 0$ and $a>0$ : $\mathbb{E}[Y]=\int_0^\infty y\,f(y)\,dy \ge \int_a^\infty y\,f(y)\,dy \ge \int_a^\infty a\,f(y)\,dy = a\,\mathbb{P}(Y\ge a)$ Why each step? First $\ge$ : we threw away the $[0,a)$ part (non-negative, only shrinks). Second $\ge$ : on $[a,\infty)$ we have $y\ge a$ . Rearrange: $\boxed{\;\mathbb{P}(Y\ge a)\le \frac{\mathbb{E}[Y]}{a}\;}$

Step 2 — Chebyshev's inequality

Apply Markov to $Y=(X-\mu)^2\ge 0$ with $a=\varepsilon^2$ : $\mathbb{P}\big((X-\mu)^2\ge \varepsilon^2\big)\le \frac{\mathbb{E}[(X-\mu)^2]}{\varepsilon^2}=\frac{\sigma^2}{\varepsilon^2}$ Since $(X-\mu)^2\ge\varepsilon^2 \iff |X-\mu|\ge\varepsilon$ : $\boxed{\;\mathbb{P}(|X-\mu|\ge\varepsilon)\le \frac{\sigma^2}{\varepsilon^2}\;}$

Step 3 — Variance of the sample mean

Why this step? Chebyshev needs the variance of $\bar X_n$ . Using independence (covariances vanish) and identical distribution: $\operatorname{Var}(\bar X_n)=\operatorname{Var}\!\Big(\tfrac1n\sum X_i\Big)=\frac{1}{n^2}\sum_{i=1}^n \operatorname{Var}(X_i)=\frac{n\sigma^2}{n^2}=\frac{\sigma^2}{n}$ Also $\mathbb{E}[\bar X_n]=\mu$ (linearity). The variance shrinks like $1/n$ — that's the engine.

Step 4 — Put it together

Apply Chebyshev to $\bar X_n$ (mean $\mu$ , variance $\sigma^2/n$ ): $\mathbb{P}(|\bar X_n-\mu|\ge\varepsilon)\le \frac{\sigma^2/n}{\varepsilon^2}=\frac{\sigma^2}{n\varepsilon^2}\xrightarrow[n\to\infty]{}0$

Figure — Law of Large Numbers — weak and strong

HOW the Strong Law works (idea, not full epsilon-grind)

The SLLN (Kolmogorov) needs only $\mathbb{E}|X_i|<\infty$ . The key tool is the Borel–Cantelli lemma:

Take $A_n=\{|\bar X_n-\mu|>\varepsilon\}$ . The crude Chebyshev bound gives $\sum_n \frac{\sigma^2}{n\varepsilon^2}$ which diverges — so a naive bound is not enough. That's exactly why SLLN is harder than WLLN. Kolmogorov uses a sharper maximal inequality (controlling the worst deviation up to $n$ at once) so the relevant series converges, then Borel–Cantelli kills all but finitely many bad events $\Rightarrow$ the path settles forever.

Worked Examples

Worked example Example 1 — Fair coin frequency

$X_i=1$ if heads, $0$ if tails, $p=\tfrac12$ . So $\mu=\tfrac12$ , $\sigma^2=p(1-p)=\tfrac14$ . Q: Bound $\mathbb{P}(|\bar X_n-\tfrac12|\ge 0.1)$ for $n=100$ . $\le \frac{\sigma^2}{n\varepsilon^2}=\frac{0.25}{100\cdot 0.01}=0.25$ Why this step? We plug straight into the WLLN bound. So at $n=100$ there's at most a 25% chance the head-fraction is off the true rate by more than 0.1. WLLN: this $\to 0$ as $n\to\infty$ . SLLN: the actual head-fraction converges to $\tfrac12$ for almost every infinite coin sequence — this is what "in the long run a fair coin gives half heads" rigorously means.

Common Mistakes (Steel-manned)

Active Recall

Recall Try before reading: state both laws and the key inequality

WLLN: $\mathbb P(|\bar X_n-\mu|>\varepsilon)\to 0$ (in probability). SLLN: $\mathbb P(\lim \bar X_n=\mu)=1$ (almost sure). Engine: $\operatorname{Var}(\bar X_n)=\sigma^2/n$ plugged into Chebyshev gives the $\frac{\sigma^2}{n\varepsilon^2}$ bound.

Recall Feynman: explain to a 12-year-old

Imagine flipping a coin and writing down the fraction of heads so far. At the start it jumps around wildly — maybe 100% heads, maybe 0%. But the more you flip, the lazier that fraction gets, slowly crawling toward one-half and staying there. Each new flip can only nudge the whole pile a tiny bit, because you're sharing it among thousands of flips. The "weak" rule says: with lots of flips you're probably near one-half. The "strong" rule says: if you flipped forever, you'd definitely land exactly on one-half and never run away again. The coin doesn't "remember" or "owe" you anything — the average calms down just because old surprises get watered down by the crowd.

Flashcards

State the Weak Law of Large Numbers.

For i.i.d.

X_i

with mean

\mu

\bar X_n\to\mu

in probability:

\forall\varepsilon>0,\ \mathbb P(|\bar X_n-\mu|>\varepsilon)\to 0

State the Strong Law of Large Numbers.

\bar X_n\to\mu

almost surely:

\mathbb P(\lim_{n\to\infty}\bar X_n=\mu)=1

What is

\operatorname{Var}(\bar X_n)

for i.i.d. with variance

\sigma^2

\sigma^2/n

(independence kills covariances).

Which inequality powers the easy WLLN proof, and what's the bound?

Chebyshev;

\mathbb P(|\bar X_n-\mu|\ge\varepsilon)\le \sigma^2/(n\varepsilon^2)

Derive Markov's inequality in one line.

\mathbb E[Y]\ge\int_a^\infty y f\,dy\ge a\int_a^\infty f\,dy=a\mathbb P(Y\ge a)

, so

\mathbb P(Y\ge a)\le\mathbb E[Y]/a

Strong vs weak: which implies which?

Almost-sure (strong)

\Rightarrow

in-probability (weak), never the reverse.

What minimal condition do Khinchin/Kolmogorov require?

Only finite mean,

\mathbb E|X|<\infty

(variance not needed).

Distribution where LLN fails and why.

Cauchy — it has no finite mean, sample means never converge.

Difference between LLN and CLT.

LLN:

\bar X_n\to\mu

(a constant). CLT: fluctuations

\sqrt n(\bar X_n-\mu)\to\mathcal N(0,\sigma^2)

Lemma behind the strong law.

Borel–Cantelli: if

\sum\mathbb P(A_n)<\infty

then

A_n

happens only finitely often (a.s.).

Why doesn't a streak of heads violate LLN?

Deviations dilute: the streak is a shrinking fraction of large

n

; no memory/compensation needed.

How many fair-coin flips so Chebyshev bound

\le0.05

for

\varepsilon=0.1

n\ge 0.25/(0.05\cdot0.01)=500

Connections

Chebyshev's Inequality — the workhorse for WLLN's rate.
Markov's Inequality — parent of Chebyshev.
Central Limit Theorem — describes the $1/\sqrt n$ fluctuations LLN ignores.
Borel–Cantelli Lemma — engine of the Strong Law.
Modes of Convergence — in probability vs almost sure vs in distribution.
Monte Carlo Methods — practical payoff of the LLN.
Gambler's Fallacy — the famous misreading of LLN.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, Law of Large Numbers ka core idea ekdum simple hai: agar tum koi random experiment baar-baar repeat karo aur unka average nikalte jao, toh wo average dheere-dheere true mean $\mu$ ke paas ja kar settle ho jaata hai. Kyun? Kyunki har individual draw ka random "shor" (noise) ek dusre ko cancel kar deta hai jab tum bahut saare add karte ho. Mathematically, $\operatorname{Var}(\bar X_n)=\sigma^2/n$ — yani jaise-jaise $n$ badhta hai, average ka bikhraav $1/n$ ki speed se ghat-ta jaata hai. Yahi engine hai.

Ab do flavours hain. Weak Law kehta hai: ek bada fixed $n$ lo, toh probability ki tum mean se $\varepsilon$ se zyada door ho — wo zero ki taraf jaati hai (convergence in probability). Strong Law zyada powerful hai: wo poore infinite path ke baare mein baat karta hai — almost har sequence ke liye $\bar X_n$ sach mein $\mu$ tak pahunch jaata hai aur phir wahin ruk jaata hai (convergence almost surely). Yaad rakho: Strong $\Rightarrow$ Weak, ulta nahi.

Proof bhi rattu maat karo — derive karo. Markov inequality se Chebyshev banao, phir $\bar X_n$ ki variance $\sigma^2/n$ daalo, aur seedha mil jaata hai bound $\mathbb P(|\bar X_n-\mu|\ge\varepsilon)\le \sigma^2/(n\varepsilon^2)$ , jo $n\to\infty$ pe zero ho jaata hai. Strong Law thoda harder hai — Borel–Cantelli lemma aur Kolmogorov ki sharper inequality lagti hai.

Sabse important warning: Gambler's fallacy mat maro. "5 heads aaye, ab tails 'due' hai" — galat! Coin ki koi memory nahi hoti. Average isliye balance hota hai kyunki purana streak bade $n$ mein ek chhota sa fraction ban jaata hai (dilution), na ki future mein koi correction force lag-ta hai. Aur LLN ko CLT se mat confuse karo: LLN batata hai average kahan land karega ( $\mu$ pe), CLT batata hai uske aas-paas wiggle ka shape kya hoga ( $1/\sqrt n$ scale pe Normal).

Law of Large Numbers — weak and strong

WHAT are we actually claiming?

HOW to derive the Weak Law (from scratch)

Step 1 — Markov's inequality

Step 2 — Chebyshev's inequality

Step 3 — Variance of the sample mean

Step 4 — Put it together

HOW the Strong Law works (idea, not full epsilon-grind)

Worked Examples

Common Mistakes (Steel-manned)

Active Recall

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability Theory & Statistics

Connections