1.3.21 · D5Probability & Statistics
Question bank — Confidence intervals
Reminder of the vocabulary these questions poke at:
- — the true population mean, a fixed unknown number (never random).
- — the sample mean, one number computed from data (random across samples).
- CI — an interval .
- coverage — the fraction of repeated intervals that would trap .
True or false — justify
The single biggest confusion in this topic hides in the word "probability." These items drill it.
A 95% CI means there is a 95% probability that lies inside this specific computed interval
False. Once numbers are plugged in, both the interval endpoints and are fixed constants, so is either in or out — the 95% describes the procedure's long-run coverage, not this one interval.
"95% confident" and "95% of my intervals capture over many repeats" mean the same thing
True. The confidence level is exactly the long-run coverage of the method; "95% confident" is shorthand for "this net-throwing procedure catches 95% of the time."
If I widen a 95% CI to a 99% CI on the same data, it gets wider
True. Higher confidence needs a larger critical value ( vs ), and margin of error scales with that value, so the interval must grow to be more sure of trapping .
Collecting 4× more data roughly halves the margin of error
True. Margin scales as , and , so quadrupling halves the width — diminishing returns are built into the square root.
A wider confidence interval always means you did something wrong
False. A wide interval honestly reports high uncertainty (small or large spread); "false precision" from a tiny sample is the actual mistake, not width.
For the same data and confidence level, the t-interval is always at least as wide as the z-interval
True. Because is itself random, the t-distribution has heavier tails, so always, giving an equal-or-wider interval.
As sample size grows, the t-interval and z-interval converge
True. As , and , so the two critical values (and intervals) become indistinguishable — this is why lets you use safely.
If two models' 95% CIs overlap, the models are statistically indistinguishable
False. Overlapping CIs are suggestive but not decisive; the correct test builds a CI for the difference — it can exclude 0 (significant) even when the individual intervals overlap.
The confidence interval tells you where 95% of individual data points lie
False. A CI is about the mean , whose spread is ; a prediction/tolerance interval for individual points uses itself and is much wider.
Spot the error
Each item contains a real reasoning slip students actually write. Name what's broken.
", unknown, so I used because it's the standard 95% value."
Error: with unknown and small , you must use ; using understates the margin and produces an interval that's too narrow (undercovers ).
"My 95% CI came out negative on the low end for a proportion, so I reported accuracy could be ."
Error: proportions are bounded in ; the normal approximation broke down near the boundary. Clip to or use a boundary-aware method like Wilson or bootstrap.
"I got a great result, so I rebuilt the CI several times on resplit data and reported the narrowest one."
Error: cherry-picking the tightest of many intervals destroys the coverage guarantee — the 95% only holds for a single, pre-committed procedure, not the min over attempts.
"The 95% CI is , so 95% of future test sets will score in this range."
Error: this confuses a CI for the mean accuracy with a prediction interval for future samples; future single-run scores vary far more than the mean does.
" came out huge because of one outlier, but I plugged it into the formula anyway."
Error: the CI assumes no gross outliers; a single point can inflate and silently invalidate the near-normal assumption. Investigate the outlier or use a robust/bootstrap method.
"I used a 95% CI, found it excluded the baseline, and concluded a 5% chance the effect is fake."
Error: the confidence level is not a per-conclusion error probability about this result; the "5%" refers to long-run false-coverage of the procedure, not the posterior probability the effect is null.
"My test set is the same 100 images I tuned hyperparameters on, but the CI still applies."
Error: the CI assumes independent, representative data; reusing tuning data breaks the "one honest random sample" premise (see Cross-validation), so the interval is optimistically biased.
Why questions
These ask for the mechanism, not the fact.
Why do we split the tail probability into on each side instead of putting it all on one side?
Because we want a two-sided interval that could miss by being too high OR too low; splitting equally makes the interval symmetric and gives the shortest interval for a symmetric distribution.
Why does the standard error use and not itself?
The CI describes the sample mean's spread, and averaging independent values shrinks variance by a factor of (via the Central Limit Theorem), so the standard deviation of is — see Standard Error.
Why do we divide by (not ) when computing the sample variance ?
Bessel's correction: we already spent one degree of freedom estimating with , so dividing by makes an unbiased estimate of instead of a systematically-too-small one.
Why does the t-distribution have heavier tails than the normal?
Because it accounts for the extra randomness of estimating by ; that added uncertainty makes extreme values more likely, fattening the tails so we don't overstate our confidence.
Why does the CLT let us build a CI even when the raw data isn't normal?
The CLT says the sample mean becomes approximately normal for large regardless of the population shape, and the CI is about — so we only need the mean's distribution to be normal, not the data's.
Why is a CI on the difference of two accuracies more informative than eyeballing two overlapping CIs?
Because the difference has its own (smaller-combined) standard error , and asking whether that interval excludes 0 is the mathematically correct significance question — this is the logic behind A/B Testing and Hypothesis Testing.
Why does a Bayesian credible interval have the "95% probability is inside" meaning that a frequentist CI does not?
A credible interval treats as a random variable with a posterior distribution, so it can literally carry that probability; a frequentist CI treats as fixed, forbidding that phrasing.
Edge cases
Every method has boundaries. These push the formulas until they crack.
What happens to the CI as ?
The margin , so the interval collapses onto the true — infinite data pins the mean exactly (though it never tells you about individual spread).
What happens to a proportion CI when (0 correct out of )?
The standard error , giving a zero-width interval — clearly absurd, so at the boundary you must switch to Wilson or bootstrap intervals that don't degenerate.
What happens if you compute a t-interval with ?
You cannot: leaves no degrees of freedom to estimate (division by ), so a one-point sample gives no information about spread and no interval exists.
If the population is perfectly constant (), what is the CI?
The margin is , so the interval is the single point — with zero variability, one sample reveals the mean exactly.
For a fixed , what happens to the margin of error as confidence ?
The critical value , so the margin explodes to infinity — being certain of trapping requires the useless interval .
When is the normal-approximation proportion CI unreliable near 0 or 1?
When or is small (roughly –); the binomial is too skewed for the symmetric normal approximation, and the interval can spill outside — use exact or Wilson methods.
If your data is heavily skewed and is moderate, is the standard CI trustworthy?
Not fully: the CLT needs a larger to normalize the mean of skewed data, so for skew you either gather more data or prefer a bootstrap CI that makes no normality assumption.
Recall Fast self-test
The interval is random, is fixed :::: State whose randomness the "95%" actually describes. Small , unknown :::: Name the exact condition forcing the t-distribution over z. A CI for the difference excluding 0 :::: State the correct way to compare two models' accuracies.