1.3.15 · D5Probability & Statistics
Question bank — Central limit theorem
Quick symbol refresher so nothing here is used before it means something:
- — the raw samples, i.i.d. (independent, same distribution).
- — the true population mean; — the true population variance.
- — the sample mean (one number per sample of size ).
- — the standard error (SE): how much wobbles around .
- — the standard normal (bell curve, mean 0, spread 1).
True or false — justify
The CLT makes the raw data become normally distributed as grows.
False. The individual never change shape — a die stays uniform forever. Only the distribution of the average approaches normal.
Bigger sample size makes the sample mean's distribution wider and more spread out.
False. SE shrinks as grows, so the distribution of gets narrower and concentrates tighter on .
The CLT requires the underlying distribution to be symmetric.
False. It works for skewed distributions (exponential, Bernoulli with ) too; skew just means you need a larger before the bell shape is convincing.
If , the sample mean is exactly normal.
False. CLT is a limit (); at any finite it is an approximation. "" is a rough rule of thumb, not a guarantee.
The CLT applies to the sum as well as the mean .
True. The sum is just ; it tends to . Mean and sum are two scalings of the same normal fact.
For the CLT to hold, the variance must be finite.
True. The proof needs (finite second moment). Distributions with infinite variance (e.g. Cauchy) break the theorem — their sample means stay Cauchy.
The CLT and the Law of Large Numbers say the same thing.
False. LLN says (a point). CLT describes the shape and spread of the wobble around that point — it is the finer, second-order statement.
Standardizing by dividing by (not by ) is a mistake.
False. It is exactly right: has SD , so you must divide by its SD, not the raw , to reach .
Spot the error
" for large ."
The variance is wrong. It should be — the mean of numbers is times less variable than a single number.
"Standard error , so it halves when I double the sample."
SE . To halve SE you must quadruple (because of the square root), not double it.
"My data is heavily skewed, so I'll just apply CLT at like usual."
For strong skew, is far too small; the approximation is still visibly lopsided. Skew slows convergence — you may need hundreds of samples.
" converges to ."
Missing the . Dividing only by gives a variable whose variance (it collapses to a spike at 0), not a fixed bell.
"The samples are correlated over time, but the CLT still applies as stated."
The classic CLT needs independence; the variance-of-a-sum step used , which fails under correlation. You'd need a dependent-data version.
"."
You forgot to standardize the endpoints. It should be .
"I know and exactly from my sample, so my CLT interval is exact."
In practice is estimated from data, which adds uncertainty — that's why small samples use the -distribution, not . See Confidence Intervals and Hypothesis Testing.
Why questions
Why does the appear in the standard error rather than ?
Variance of the sum grows like , and divides by , giving variance ; SD is the square root of variance, so .
Why is CLT the engine behind Confidence Intervals and Hypothesis Testing?
Both need the sampling distribution of ; CLT hands us a known normal shape, so we can convert probabilities into -scores and thresholds.
Why does a mini-batch gradient in Stochastic Gradient Descent behave like a noisy version of the true gradient?
Each batch gradient is a sample mean over batch elements, so by CLT it is — unbiased with noise shrinking as batch size grows.
Why must the terms be identically distributed (not just independent) in the basic CLT?
The proof raises one MGF to the -th power, ; that single shared MGF only exists if every has the same distribution.
Why do we standardize before taking the limit at all?
Un-standardized, collapses onto (LLN) — there's nothing left to see. Rescaling by "zooms in" so the residual wobble stabilizes into a fixed bell.
Why can Bootstrap Methods and Monte Carlo Methods rely on averaging many resamples?
Their estimates are averages, so their sampling error is normal with SE — CLT tells us how fast the estimate settles.
Why doesn't CLT help a Maximum Likelihood Estimation problem with an infinite-variance model?
No finite means the standardizing SE is undefined and the Taylor step collapses; the sampling distribution simply doesn't converge to normal.
Edge cases
What happens to if every is a constant (zero variance)?
always; SE , so the "distribution" is a single spike. CLT is vacuously true but degenerate — nothing to standardize.
What if ?
Then , so the sample-mean distribution is just the raw distribution — no normalizing has happened. CLT only bites for growing .
The data is already — does CLT add anything?
The mean is exactly for every , no approximation needed. CLT's promise is already met at because sums of normals stay normal.
Bernoulli with (extremely rare event) — is enough for normality?
No. With near-zero most samples are 0, so the sum is Poisson-like and skewed; you need and both comfortably large (a common check ) before the bell is trustworthy.
Two very heavy-tailed but finite-variance sources: does CLT still guarantee normality?
Yes eventually, but heavy tails slow it dramatically — the Bias-Variance Tradeoff and Sample Size Calculation both feel this, since the required can be huge.
If I average dependent samples that are positively correlated, what goes wrong with SE?
The true SE is larger than (positive covariance adds to the variance-of-sum), so a naive CLT interval is over-confident — too narrow.
Recall One-line self-test
The mean is exactly normal for any when the data itself is ::: normal (sums of normals are normal). SE shrinks by what factor when goes ? ::: by a factor of 2, since . The one moment condition CLT cannot do without is ::: finite variance ().