Law of large numbers: Worked examples — AI-ML deep dive

Parent: Law of large numbers · This child is the drill page. We build a grid of every situation the Law of Large Numbers (LLN) can hand you, then solve one example per grid cell. Nothing here assumes you memorised the parent — every symbol is re-earned as it appears.

Before we touch a single example, one promise: every letter we write gets a plain-English meaning and (where it helps) a picture. Let's list the cast of characters once.

Definition The characters (read this first)

$X_{i}$ — the outcome of the $i$ -th repeat of an experiment. " $i$ " is just a counter: $X_{1}$ is the first result, $X_{2}$ the second, and so on. Think one dice roll, one coin flip, one measured height.
$μ$ (Greek "mu") — the expected value, the true long-run average of a single $X_{i}$ . See Expected Value. Picture the balance point of the outcome histogram.
$σ^{2}$ (sigma-squared) — the variance, how spread-out one $X_{i}$ is. See Variance and Standard Deviation. $σ$ itself (no square) is the standard deviation, the typical distance from $μ$ .
$\overset{ˉ}{X}_{n} = \frac{1}{n} \sum_{i = 1}^{n} X_{i}$ — the sample mean, i.e. add up your first $n$ results and divide by $n$ . The bar means "average of", the subscript $n$ means "using $n$ samples".
$ϵ$ (epsilon) — a small positive "how close do I demand" tolerance, e.g. $0.01$ .
$δ$ (delta) — the "how much failure I'll tolerate" probability budget, e.g. $0.05$ .
i.i.d. — Independent and Identically Distributed: every $X_{i}$ is drawn the same way (identical) and no draw influences another (independent).

The single tool we lean on is the Chebyshev Inequality. Let's earn it in plain words before we use it anywhere.

Intuition Why the Chebyshev bound is true (words, not symbols)

To state the rule in general we need a name for "any random quantity we care about". Call it $Y$ — just a stand-in letter for an arbitrary random variable (it could be one coin flip, or a whole sample mean; we'll say which each time). It has its own mean $μ$ and variance $σ^{2}$ .

Variance $σ^{2}$ is, by its very definition, the average of the squared distance of $Y$ from its mean. Now ask: how much of that average can come from outcomes that land far (distance $\geq ϵ$ ) from the mean? Each such far outcome contributes at least $ϵ^{2}$ to the squared-distance average. So if a fraction $p$ of the probability sits that far out, those outcomes alone already pile up at least $p \cdot ϵ^{2}$ into the average. But the whole average is only $σ^{2}$ . Therefore $p \cdot ϵ^{2} \leq σ^{2}$ , i.e. $p = P (∣ Y - μ ∣ \geq ϵ) \leq \frac{σ ^{2}}{ϵ ^{2}} .$ That's Chebyshev for a single variable $Y$ . To get our sample-mean version, apply it with $Y$ being the sample mean $\overset{ˉ}{X}_{n}$ , whose variance (independence makes variances add, then the $\frac{1}{n}$ scales them by $\frac{1}{n ^{2}}$ ) is $σ^{2} / n$ . Substitute and you get the bound below. We will re-say this "far mass must fit inside the variance" sentence at each example that uses it.

The scenario matrix

Every LLN problem lives in one of these cells. The examples below are labelled by cell.

Cell	Scenario class	What makes it tricky	Example
A	Bounded 0/1 outcome (Bernoulli)	variance $= p (1 - p)$	Ex 1
B	Solve for required $n$	invert the bound	Ex 2
C	Large-spread continuous variable	big $σ$ , real units	Ex 3
D	Zero-variance / degenerate input	$σ^{2} = 0$ edge case	Ex 4
E	Limiting behaviour ( $n \to \infty$ and $ϵ \to 0$ )	which wins the race	Ex 5
F	Monte Carlo estimate of an integral	random variable is an indicator	Ex 6
G	Word problem / "is my coin rigged?"	interpret, don't panic	Ex 7
H	Exam twist: heavy-tail / infinite variance	LLN can fail	Ex 8

Ex 1 — Cell A: bounded 0/1 outcome

The figure above shows the picture behind LLN: light grey lines are 40 individual random runs of the running head-proportion, each jittering wildly at small $n$ ; the thick cyan line is their average, hugging $μ = 0.5$ ever more tightly as $n$ grows. The two dashed amber curves are the $μ \pm σ / n$ "error band" (the standard-error funnel) that narrows like $1/ n$ — the funnel every run gets sucked into. Keep this funnel in mind for every example below.

Worked example Fair coin, fixed

$n$ Flip a fair coin $n = 10, 000$ times. Let $X_{i} = 1$ for heads, $0$ for tails. Bound the probability that the head-proportion $\overset{ˉ}{X}_{n}$ misses $0.5$ by at least $ϵ = 0.01$ .

Forecast: guess now — is the bound closer to $1%$ , $25%$ , or $90%$ ?

Step 1. Find $μ$ . Since $X_{i}$ is $1$ half the time, $μ = 0.5$ . Why this step? Chebyshev needs the true mean; for a fair coin it's the heads-probability.

Step 2. Find $σ^{2}$ . For a 0/1 variable, $X_{i}^{2} = X_{i}$ (because $0^{2} = 0, 1^{2} = 1$ ), so $E [X_{i}^{2}] = E [X_{i}] = 0.5$ , giving $σ^{2} = 0.5 - 0. 5^{2} = 0.25$ . Why this step? Variance $= E [X^{2}] - μ^{2}$ ; the algebra collapses nicely for 0/1 outcomes.

Step 3. Plug in ("far mass must fit inside the variance"): $P (∣ \overset{ˉ}{X}_{n} - 0.5∣ \geq 0.01) \leq \frac{0.25}{10000 \cdot ( 0.01 ) ^{2}} = \frac{0.25}{1} = 0.25.$ Why this step? Direct substitution into the one bound.

Verify: Units are pure probability (dimensionless), and $0.25 \leq 1$ — a valid probability. The sharper CLT answer is $\approx 0.046$ , comfortably below our $0.25$ , so our bound is consistent (it must over-estimate). ✓ Read the number honestly: $0.25$ is only a worst-case ceiling — up to a 1-in-4 chance — not a claim that a miss is rare. It merely says "no worse than 25%"; the true chance (CLT) is the genuinely small $\approx 4.6%$ . In the funnel figure, the standard-error band at $n = 10, 000$ has shrunk to width $\pm 0.005$ , which is why the real miss chance is so much smaller than the crude bound.

Ex 2 — Cell B: solve for the required $n$

Worked example How many people to pin down an average height?

Heights have $μ = 170$ cm and $σ = 10$ cm. How large must $n$ be so the sample mean is within $ϵ = 1$ cm of $170$ with probability $\geq 0.95$ ?

Forecast: more or fewer than 1000 people?

Step 1. Translate the demand. "Within 1 cm with prob $\geq 0.95$ " means the miss probability is $\leq δ = 0.05$ : $P (∣ \overset{ˉ}{X}_{n} - 170∣ \geq 1) \leq 0.05$ . Why this step? Chebyshev bounds the miss, not the hit; convert first. Here $δ$ is our failure budget.

Step 2. Force the guaranteed worst case (the "far mass" bound) below the budget: $\frac{σ ^{2}}{n ϵ ^{2}} \leq δ \Rightarrow \frac{100}{n \cdot 1} \leq 0.05.$ Why this step? If even the over-estimate is $\leq 0.05$ , the true miss chance certainly is.

Step 3. Solve: $n \geq 100/0.05 = 2000$ . Why this step? Algebra — isolate $n$ .

Verify: Plug $n = 2000$ back: $100/ (2000 \cdot 1) = 0.05$ exactly. ✓ (CLT would need only ~385; Chebyshev is deliberately conservative — see Confidence Intervals.)

Ex 3 — Cell C: large-spread continuous variable

Worked example Noisy sensor readings

A sensor reports a value with true mean $μ = 20$ and standard deviation $σ = 8$ units. You average $n = 64$ readings. Bound the chance the average is off by $\geq 2$ units.

Forecast: below or above $50%$ ?

Step 1. Identify: $σ^{2} = 8^{2} = 64$ , $n = 64$ , $ϵ = 2$ . Why this step? Chebyshev works for any finite-variance distribution — we never needed the sensor's exact shape.

Step 2. Get a feel for the scale with the standard error: $SE = σ / n = 8/ 64 = 8/8 = 1$ . So the average typically wobbles about $1$ unit, and our target $ϵ = 2$ is "2 standard errors out". Why this step? The standard error is only a feel-for-the-scale quantity (the funnel half-width from Ex 1); it tells us $ϵ = 2$ is a fairly far miss, but it is not itself a probability. To turn "how far" into "how likely" we must go back to the guaranteed Chebyshev bound in Step 3 — that is the switch, and here is why we make it.

Step 3. Turn distance into a probability bound ("far mass must fit inside the variance"): $P (∣ \overset{ˉ}{X}_{n} - 20∣ \geq 2) \leq \frac{σ ^{2}}{n ϵ ^{2}} = \frac{64}{64 \cdot 2 ^{2}} = \frac{64}{256} = 0.25.$ Why this step? Only the Chebyshev bound converts a spread ( $SE$ ) into a guaranteed chance-of-missing; the standard error alone cannot.

Verify: $ϵ = 2$ is exactly $2$ standard errors ( $2/1$ ). Chebyshev at "2 SE" gives $1/ 2^{2} = 0.25$ — matches. ✓ Units: value units cancel inside the probability. ✓

Ex 4 — Cell D: the degenerate / zero-variance input

Worked example A "constant" experiment

Every $X_{i}$ equals exactly $7$ (a broken thermometer stuck at 7). Then $μ = 7$ , $σ^{2} = 0$ . What does the bound say about $P (∣ \overset{ˉ}{X}_{n} - 7∣ \geq ϵ)$ for any $ϵ > 0$ ?

Forecast: does the average ever wander?

Step 1. Compute $σ^{2}$ . All values identical $\Rightarrow$ no spread $\Rightarrow$ $σ^{2} = 0$ . Why this step? Variance measures deviation from $μ$ ; with no deviation it's $0$ .

Step 2. Plug in: $P (∣ \overset{ˉ}{X}_{n} - 7∣ \geq ϵ) \leq \frac{0}{n ϵ ^{2}} = 0.$ Why this step? Zero numerator forces the bound to $0$ for every $n$ , even $n = 1$ . (In the funnel picture, the band has zero width — a single flat line.)

Step 3. Interpret: the average is glued to $7$ from the very first sample. Why this step? This is the trivial edge case where LLN's convergence is instantaneous.

Verify: $\overset{ˉ}{X}_{n} = \frac{1}{n} (7 + 7 + \dots + 7) = \frac{7 n}{n} = 7$ literally. A probability of being off is genuinely $0$ . ✓

Ex 5 — Cell E: limiting behaviour, who wins the race?

The plot shows the Chebyshev bound $σ^{2} / (n ϵ^{2})$ as $n$ grows, for three ways of choosing $ϵ$ . The cyan curve (case a, fixed $ϵ = 0.1$ ) slides down to the axis: error probability vanishes. The amber dashed line (case c, $ϵ = 1/ n$ ) sits flat at height $1$ : the bound never improves. The short white arrow marks case (b), where shrinking $ϵ$ at fixed $n$ blows the bound upward off the top of the frame. Read the curves alongside the algebra.

Worked example Shrink

$n$ vs shrink $ϵ$ Using $σ^{2} = 1$ : (a) fix $ϵ = 0.1$ and let $n \to \infty$ ; (b) fix $n = 100$ and let $ϵ \to 0$ ; (c) let $ϵ = 1/ n$ with $n \to \infty$ . What happens to the bound $σ^{2} / (n ϵ^{2})$ in each?

Forecast: which case does NOT go to zero?

Step 1 (a). $\frac{1}{n \cdot 0. 1 ^{2}} = \frac{100}{n} \to 0$ as $n \to \infty$ . Why this step? This IS the Weak Law — hold tolerance fixed, grow samples, error probability vanishes (cyan curve dropping toward the axis).

Step 2 (b). $\frac{1}{100 \cdot ϵ ^{2}} \to \infty$ as $ϵ \to 0$ . Why this step? Demanding zero error with a fixed sample gives a useless (>1) bound — the white arrow shooting off the top. LLN never promises exactness at finite $n$ .

Step 3 (c). $ϵ = 1/ n \Rightarrow \frac{1}{n \cdot ( 1/ n ) ^{2}} = \frac{1}{n \cdot ( 1/ n )} = 1.$ Why this step? When you shrink the tolerance at exactly the natural $1/ n$ rate, the bound stalls at $1$ (the flat amber line) — a tie. This is precisely the scale Central Limit Theorem lives on.

Verify: (a) $100/ n \to 0$ ✓; (c) plug $n = 25$ : $1/ (25 \cdot (1/5)^{2}) = 1/ (25 \cdot 1/25) = 1$ ✓.

Ex 6 — Cell F: Monte Carlo integral

The dart picture shows the unit square with the amber curve $y = e^{- x^{2}}$ ; cyan dots landed under the curve ( $Z_{i} = 1$ ), white dots landed above ( $Z_{i} = 0$ ). The fraction of cyan dots estimates the shaded area, which is exactly the integral. As we throw more darts the cyan fraction locks onto the true value — LLN in a picture.

Worked example Estimate

$\int_{0}^{1} e^{- x^{2}} d x$ by dart-throwing Throw random darts into the unit square. $X_{i}, Y_{i} \sim Uniform (0, 1)$ ; set $Z_{i} = 1$ if the dart lands under the curve $y = e^{- x^{2}}$ , else $0$ . Estimate the integral, then find how many darts guarantee the estimate is within $ϵ = 0.001$ of the truth with failure budget $δ = 0.05$ . See Monte Carlo Methods.

Forecast: thousands or hundreds of thousands of darts?

Step 1. Why does the average of $Z_{i}$ estimate the integral? The fraction of darts under the curve is the area under the curve, which is exactly $\int_{0}^{1} e^{- x^{2}} d x \approx 0.7468$ . So $μ = E [Z_{i}] = \int_{0}^{1} e^{- x^{2}} d x$ . Why this step? LLN promises $\overset{ˉ}{Z}_{n} \to E [Z_{i}]$ ; we engineered $Z_{i}$ so its mean IS the answer.

Step 2. Bound the spread. $Z_{i}$ is a 0/1 variable, so $σ^{2} = p (1 - p) \leq 0.25$ (max at $p = 0.5$ ). Why this step? Worst-case variance gives a safe sample count without knowing $p$ exactly.

Step 3. Use the real Chebyshev sample-size condition — not just the standard error. We need the miss-probability below the budget:

\;\Rightarrow\; n \ge \frac{\sigma^2}{\delta\,\epsilon^2}.$$ *Why this step?* This is the honest guarantee. The standard error $\sigma/\sqrt n$ (defined earlier — the typical wobble) only describes the estimator's usual jitter, not a *probability* that it stays within $\epsilon$; only Chebyshev gives that probability. **Step 4.** Plug in $\sigma^2=0.25$, $\delta=0.05$, $\epsilon=0.001$: $$n\ge\frac{0.25}{0.05\cdot(0.001)^2}=\frac{0.25}{0.05\cdot 10^{-6}}=5{,}000{,}000.$$ *Why this step?* Arithmetic. Note it is far larger than a naive "standard error $<0.001$" count of $250{,}000$ — because a *probabilistic* guarantee is much stronger than merely shrinking the standard deviation. **Verify:** $0.25/(0.05\cdot 10^{-6}) = 0.25/(5\times10^{-8}) = 5\times10^6$ ✓. As a sanity check the standard error at $n=5\times10^6$ is $0.5/\sqrt{5\times10^6}\approx 2.2\times10^{-4}$, safely under $\epsilon$, consistent with the stronger bound. ✓ And $\int_0^1 e^{-x^2}dx=0.7468\ldots$ ✓.

Ex 7 — Cell G: the "is my coin rigged?" word problem

Worked example 60 heads out of 100 — panic?

You flip a coin $100$ times and see $60$ heads. Is the coin biased?

Forecast: damning evidence, or ordinary luck?

Step 1. Assume fairness and find the standard error of the head-proportion. With $σ^{2} = 0.25$ per flip, $SE = σ / n = 0.25 / 100 = 0.5/10 = 0.05$ . Why this step? We test how surprising $0.60$ is relative to natural fluctuation — the funnel half-width at $n = 100$ .

Step 2. Measure the distance: $(0.60 - 0.50) /0.05 = 0.10/0.05 = 2$ standard errors. Why this step? "How many standard errors out" tells us how far into the funnel's edge we are.

Step 3. A 2-standard-error deviation is not extreme — it happens fairly often. LLN says fluctuations shrink as $n$ grows but never vanish, so a single 60/100 is weak evidence. Why this step? Guards against the gambler's-fallacy panic; see Confidence Intervals.

Verify: Chebyshev at 2 standard errors: $P (∣ \overset{ˉ}{X} - 0.5∣ \geq 0.10) \leq 1/ 2^{2} = 0.25$ — up to a 1-in-4 chance, nowhere near "impossible". ✓ So 60/100 is unremarkable for a fair coin.

Ex 8 — Cell H: exam twist, LLN can FAIL

Worked example When there is no finite mean

Suppose each $X_{i}$ follows a Cauchy distribution — a bell-ish curve with tails so heavy that $μ$ and $σ^{2}$ are undefined (the defining integrals diverge). What does LLN say about $\overset{ˉ}{X}_{n}$ ?

Forecast: does averaging help here?

Step 1. Check the hypotheses. Every LLN theorem in the parent begins "with finite expected value $μ$ ". Cauchy has no finite mean. Why this step? A theorem's guarantee only applies when its assumptions hold — always check them.

Step 2. Consequence: $\overset{ˉ}{X}_{n}$ does not settle down. Remarkably, for Cauchy the sample mean of $n$ values has the same distribution as a single value — averaging buys you nothing. Why this step? Shows the $1/ n$ shrinkage was never free; it required finite $σ^{2}$ . (No funnel forms — the band never narrows.)

Step 3. Lesson for ML: before trusting a Monte Carlo average or an Stochastic Gradient Descent estimate, confirm the quantity has finite mean/variance. Why this step? Heavy-tailed losses or rewards can silently break convergence.

Verify: No numeric claim — this is a hypothesis-checking counterexample. The takeaway: LLN's promise is conditional. ✓

Recall Quick self-test

Which cell has $σ^{2} = 0$ ? ::: Cell D — the constant/degenerate experiment (bound is $0$ for all $n$ ). To make error 10× smaller, samples must grow by what factor? ::: $100 \times$ , because error scales as $1/ n$ . Why can Chebyshev handle any distribution shape? ::: It only uses $μ$ and $σ^{2}$ , not the full shape. What is the standard error and how does it differ from the Chebyshev bound? ::: SE $= σ / n$ is the typical wobble (funnel half-width); the Chebyshev bound is a guaranteed probability of staying within $ϵ$ — a stronger claim. When does LLN fail? ::: When the mean/variance are not finite (e.g. Cauchy) — Cell H.

Connections worth chasing: Bias-Variance Tradeoff, Bootstrap Sampling, यही टॉपिक हिंदी में.