Confidence intervals: Worked examples — AI-ML deep dive

This is a worked-examples deep dive for Confidence intervals. The parent note built the formulas; here we hammer them against every kind of input a confidence-interval problem can throw at you — big samples, tiny samples, proportions, differences of proportions, differences of means, one-sided questions, and the degenerate edges where the formula almost breaks.

Figure s01 (read this first): the black curve is the sampling distribution of the mean; the red bar is the confidence-interval "net" of half-width $E$ thrown around the observed $\overset{x}{ˉ}$ (black dot). The true $μ$ (black cross) is fixed — the net is what moves from sample to sample. Every example below just resizes this red bar.

The scenario matrix

Before working anything, let us enumerate every case class a CI problem can be. Rather than a dense grid, read the decision tree below — it is the same content in visual-first form, and each leaf names the example that fills it.

Figure s02 (the map of this whole page): follow the arrows to land on your case. The red leaf ("small $n$ ⇒ use $t$ ") is the one beginners most often get wrong. Each leaf is tagged with its example number.

The dials for each cell:

Confidence dial: $z_{0.025} = 1.96$ (95%), $z_{0.005} = 2.576$ (99%), $z_{0.05} = 1.645$ (90% two-sided or 95% one-sided).
Spread dial: $σ$ if known, else $s$ (sample std, with the $n - 1$ from Confidence intervals).
Shrink dial: $n$ .

Worked Examples

Cell A — mean, σ known

Worked example Example 1 — sensor calibration

A temperature sensor is known from the factory to have measurement noise with standard deviation $σ = 0. 5^{\circ}$ C. You take $n = 25$ readings and get $\overset{x}{ˉ} = 21. 0^{\circ}$ C. Build a 95% CI for the true temperature.

Forecast: With 25 readings and a small known noise, do you expect the interval to be roughly $\pm 0.02$ , $\pm 0.2$ , or $\pm 2$ degrees? Guess before reading on.

Step 1 — pick the tool. $σ$ is known, so use $z$ , not $t$ . Why this step? The $t$ -distribution only exists to pay for our uncertainty about $σ$ . Here there is none, so $z$ is exact.

Step 2 — standard error. $S E = \frac{σ}{n} = \frac{0.5}{25} = \frac{0.5}{5} = 0.1$ Why this step? $S E$ is the spread of the sample mean, not of one reading. Averaging 25 values makes the mean $25 = 5 \times$ tighter than a single reading.

Step 3 — margin of error. $E = z_{0.025} \cdot S E = 1.96 \times 0.1 = 0.196$ Why this step? $z_{0.025} = 1.96$ is the point of the standard normal that leaves 2.5% in each tail (5% total = the $α$ we allow to miss).

Step 4 — assemble. The interval is $\overset{x}{ˉ}$ minus $E$ on the left and $\overset{x}{ˉ}$ plus $E$ on the right: $CI_{95%} = 21.0 \pm 0.196 = [21.0 - 0.196, 21.0 + 0.196] = [20.804, 21.196]$ Why this step? $E$ is the half-width; we lay it off once to the left and once to the right of the point estimate, giving a total length $2 E = 0.392$ .

Verify: Half-width $0.196$ is between $0.02$ and $2$ — matches the middle forecast. Full length $= 2 \times 0.196 = 0.392$ , and $21.196 - 20.804 = 0.392$ ✓. Units are °C throughout ✓. Sanity: 1.96 SEs is a hair under 2 SEs, so $\approx 2 \times 0.1 = 0.2$ ✓.

Cell B — mean, σ unknown, large n

Worked example Example 2 — model latency, big log

You log the inference latency of a model on $n = 400$ requests. Sample mean $\overset{x}{ˉ} = 52$ ms, sample std $s = 10$ ms. 95% CI for the true mean latency?

Forecast: With 400 samples, will you use $t$ or $z$ ? Will the interval be wider or narrower than Example 1's $\pm 0.196$ ?

Step 1 — tool choice. $σ$ unknown, but $n = 400 \geq 30$ , so $s \approx σ$ and we may use $z$ . Why this step? For large $n$ , $t_{399}$ is visually indistinguishable from $N (0, 1)$ (heavier tails vanish). Using $z$ costs almost nothing.

Step 2 — standard error. $S E = \frac{s}{n} = \frac{10}{400} = \frac{10}{20} = 0.5 ms$ Why this step? We swap the unknown $σ$ for the sample std $s$ and, exactly as in Example 1, divide by $n$ to get the spread of the mean rather than of a single request — 400 requests make the mean $400 = 20 \times$ tighter than one request.

Step 3 — margin & interval. $E = 1.96 \times 0.5 = 0.98, CI_{95%} = 52 \pm 0.98 = [51.02, 52.98] ms$ Why this step? Same recipe as Cell A — we multiply the 95% critical value 1.96 by the standard error $S E = 0.5$ to turn "spread of the mean" into a half-width, then lay that half-width off to the left and right of $\overset{x}{ˉ} = 52$ . Multiplying by 1.96 (not, say, 1) is what pins the coverage at exactly 95%; centering on $\overset{x}{ˉ}$ is what makes the interval an honest statement about the unknown $μ$ .

Verify: $t_{0.025, 399} \approx 1.966$ — replacing 1.96 by 1.966 changes $E$ from $0.980$ to $0.983$ , a $0.3%$ difference. Negligible, confirming the "use $z$ " decision ✓. Units ms ✓.

Cell C — mean, σ unknown, small n

Worked example Example 3 — five A/B experiment lifts

A cautious team ran the same experiment $n = 5$ times, getting conversion lifts (in %): $2.0, 3.5, 1.0, 4.0, 2.5$ . Build a 90% CI for the true lift.

Forecast: With only 5 points, will $t$ be much bigger than $z = 1.645$ ? Guess whether the interval spans zero.

Step 1 — sample mean. $\overset{x}{ˉ} = \frac{2.0 + 3.5 + 1.0 + 4.0 + 2.5}{5} = \frac{13.0}{5} = 2.6$

Step 2 — sample std with Bessel's $n - 1$ . Deviations from 2.6: $- 0.6, 0.9, - 1.6, 1.4, - 0.1$ . Squares: $0.36, 0.81, 2.56, 1.96, 0.01$ , summing to $5.70$ . $s = \frac{5.70}{5 - 1} = 1.425 = 1.1937$ Why $n - 1$ ? We already spent one degree of freedom (see the [!definition] above) pinning $\overset{x}{ˉ}$ ; dividing by $df = 4$ (not 5) un-biases $s^{2}$ . See Confidence intervals.

Step 3 — standard error. $S E = \frac{s}{5} = \frac{1.1937}{2.2361} = 0.5338$

Step 4 — critical value. With $df = n - 1 = 4$ and 90% two-sided ⇒ $t_{0.05, 4} = 2.132$ . Why $t$ and why so big? $n < 30$ and $σ$ unknown. With only 4 degrees of freedom the tails are fat, so 2.132 ≫ 1.645 — the interval must be wider to stay honest.

Step 5 — interval. $E = 2.132 \times 0.5338 = 1.138, CI_{90%} = 2.6 \pm 1.138 = [1.462, 3.738]$

Verify: Interval does not include 0 ⇒ lift is significant at 90%. Compare: had we wrongly used $z = 1.645$ , $E = 0.878$ , giving $[1.72, 3.48]$ — falsely narrower. The $t$ correction widened it by $\approx 30%$ ✓.

Figure s03 (why small $n$ demands $t$ ): the black curve is the standard normal; the red curve is the $t$ -distribution with $df = 4$ . Notice the red curve's fatter tails — that is why its 90% cutoff (red dashed, at 2.132) sits further out than the normal's (black dashed, at 1.96). Fatter tails = wider net = honest extra caution for tiny samples. This is the picture behind Example 3, Step 4.

Cell D — a proportion

Worked example Example 4 — classifier accuracy

A classifier is right on 87 of 100 test cases. 95% CI for true accuracy?

Forecast: Will the half-width be near $\pm 0.03$ , $\pm 0.07$ , or $\pm 0.15$ ?

Step 1 — point estimate. $\overset{p}{^} = 87/100 = 0.87$ .

Step 2 — proportion standard error. $S E = \frac{p ^ ( 1 - p ^ )}{n} = \frac{0.87 \times 0.13}{100} = 0.0011310 = 0.033630$ Why this SE? A "correct/incorrect" count is binomial with variance $n p (1 - p)$ ; a proportion divides by $n$ twice, giving variance $p (1 - p) / n$ . See Standard Error.

Step 3 — interval. $E = 1.96 \times 0.033630 = 0.065915, CI_{95%} = 0.87 \pm 0.0659 = [0.8041, 0.9359]$ Why this step? Identical recipe to the mean cases — the only thing that changed is which $S E$ formula we fed in. The 1.96 (95%) and the "lay off half-width both sides" logic are unchanged.

Verify: Half-width $0.066$ ≈ the $\pm 0.07$ forecast ✓. Both endpoints lie in $[0, 1]$ (valid probabilities) ✓. Rule-of-thumb check $n p = 87 \geq 5$ and $n (1 - p) = 13 \geq 5$ , so the normal approximation is legitimate ✓.

Worked example Example 4b — the extreme-proportion edge (Wilson / Jeffreys)

A model is right on all 10 of 10 test cases: $\overset{p}{^} = 10/10 = 1.0$ . Build a 95% CI.

Forecast: The naïve formula gives $S E = 1 \cdot 0/10 = 0$ , so it claims the interval is $[1, 1]$ — "we are certain accuracy is exactly 100%". Does 10-for-10 really prove perfection?

Step 1 — spot why the normal formula collapses. When $\overset{p}{^} = 0$ or $\overset{p}{^} = 1$ , $\overset{p}{^} (1 - \overset{p}{^}) = 0$ , so $S E = 0$ and the interval degenerates to a point. Why this step? The normal approximation needs $n \overset{p}{^} \geq 5$ and $n (1 - \overset{p}{^}) \geq 5$ ; here $n (1 - \overset{p}{^}) = 0$ , so the approximation is simply invalid, not exact.

Step 2 — use the Wilson interval instead. The Wilson score interval never gives zero width at the boundary and always stays inside $[0, 1]$ : $center = \frac{p ^ + \frac{z ^{2}}{2 n}}{1 + \frac{z ^{2}}{n}}, half-width = \frac{z}{1 + \frac{z ^{2}}{n}} \frac{p ^ ( 1 - p ^ )}{n} + \frac{z ^{2}}{4 n ^{2}}$ With $\overset{p}{^} = 1$ , $n = 10$ , $z = 1.96$ (so $z^{2} = 3.8416$ ): center $= \frac{1 + 3.8416/20}{1 + 3.8416/10} = \frac{1.19208}{1.38416} = 0.8613$ , and half-width $= \frac{1.96}{1.38416} 0 + \frac{3.8416}{400} = 1.41603 \times 0.098 = 0.1388$ , giving $[0.7224, 0.9999]$ . Why this step? Wilson adds an imaginary $z^{2} /2$ successes and $z^{2} /2$ failures, nudging the estimate off the boundary so uncertainty is honestly reported.

Verify: Wilson lower bound $0.7224 < 1$ — it correctly refuses to claim certainty from 10 trials ✓; upper bound $0.9999 \leq 1$ (can't exceed 100%) ✓. (The Jeffreys interval, based on a Beta prior, behaves similarly at the edges — mention it as the Bayesian cousin, see Bayesian Credible Intervals.)

Cell E — difference of two proportions

Worked example Example 5 — is Model B really better?

Model A: 82% correct, reported margin $\pm 3%$ (95%). Model B: 85% correct, margin $\pm 4%$ . Is B significantly better?

Forecast: Their intervals overlap. Does overlap alone prove "no difference"? (Trap!)

Step 1 — recover each SE from its margin. A 95% margin is $1.96 \times S E$ , so $S E = E /1.96$ . $S E_{A} = \frac{0.03}{1.96} = 0.015306, S E_{B} = \frac{0.04}{1.96} = 0.020408$ Why this step? The problem hands us margins, not standard errors. Since a margin is built from the SE by multiplying by 1.96, we divide by 1.96 to run that construction backwards and recover the raw uncertainty of each model.

Step 2 — SE of the difference (add variances). For independent estimates, variances add: $S E_{diff} = S E_{A}^{2} + S E_{B}^{2} = 0.01530 6^{2} + 0.02040 8^{2} = 0.00065598 = 0.025612$ Why add variances? Uncertainty in a difference is the Pythagorean sum of the two independent uncertainties, never the plain sum. (This is a variance-sum SE for the difference — do not call it a "pooled SE," which is a different construction that assumes a common variance.) See A/B Testing.

Step 3 — CI of the difference. $\overset{p}{^}_{B} - \overset{p}{^}_{A} = 0.85 - 0.82 = 0.03$ . $E = 1.96 \times 0.025612 = 0.050200, CI_{95%} = 0.03 \pm 0.0502 = [- 0.0202, 0.0802]$ Why this step? Same universal recipe once more: point estimate (here the difference $0.03$ ) $\pm$ (critical value $\times$ the difference's SE). We center on the difference because that is the quantity the question asks about.

Verify: The interval contains 0, so the difference is not significant — despite B's point estimate being higher. Lesson: overlapping individual CIs is a weak test; the difference CI is the correct one, and here it confirms "not conclusive" ✓. Related: Hypothesis Testing.

Cell E2 — difference of two means

Worked example Example 5b — did the new optimizer lower loss?

Old optimizer: $n_{1} = 36$ runs, mean loss $\overset{x}{ˉ}_{1} = 0.50$ , sample std $s_{1} = 0.12$ . New optimizer: $n_{2} = 36$ runs, mean loss $\overset{x}{ˉ}_{2} = 0.44$ , sample std $s_{2} = 0.09$ . 95% CI for the drop in mean loss $μ_{1} - μ_{2}$ ?

Forecast: The point drop is $0.06$ . Will the CI clear 0 (real improvement) or straddle it?

Step 1 — SE of each mean. Large $n$ ( $\geq 30$ ), $σ$ unknown ⇒ Cell B logic, use $z$ with $s$ . $S E_{1} = \frac{0.12}{36} = 0.02, S E_{2} = \frac{0.09}{36} = 0.015$ Why this step? Each mean has its own standard error, computed exactly as in Example 2.

Step 2 — SE of the difference (add variances). $S E_{diff} = S E_{1}^{2} + S E_{2}^{2} = 0.0 2^{2} + 0.01 5^{2} = 0.000625 = 0.025$ Why the same rule as Ex 5? Two independent means differ with a Pythagorean-sum uncertainty, identical in spirit to the two-proportion case — the SE formula fed in just came from means, not proportions.

Step 3 — CI of the difference. $E = 1.96 \times 0.025 = 0.049, CI_{95%} = 0.06 \pm 0.049 = [0.011, 0.109]$ Why this step? Center on the observed drop $\overset{x}{ˉ}_{1} - \overset{x}{ˉ}_{2} = 0.06$ ; lay off the half-width $0.049$ both sides.

Verify: Interval $[0.011, 0.109]$ excludes 0, so the loss drop is significant at 95% ✓ — unlike Example 5, this difference clears zero. Units are loss units throughout ✓.

Cell F — one-sided bound

Worked example Example 6 — a floor on accuracy

Same classifier as Ex 4 ( $\overset{p}{^} = 0.87$ , $S E = 0.033630$ , $n = 100$ ). Your deployment gate only asks: "Are we at least 95% confident accuracy exceeds some floor $L$ ?" Find $L$ .

Forecast: Should the one-sided critical value be larger or smaller than 1.96?

Step 1 — put all $α$ in one tail. For a one-sided 95% lower bound, all 5% goes to the left tail, so the critical value is $z_{0.05} = 1.645$ , not 1.96. Why smaller? We only guard against being wrong on one side, so we can afford a less extreme cutoff — the one-sided bound is tighter than each side of a two-sided interval.

Step 2 — lower confidence bound. $L = \overset{p}{^} - 1.645 \times S E = 0.87 - 1.645 \times 0.033630 = 0.87 - 0.055322 = 0.814678$ Why this step? A one-sided lower bound only subtracts the half-width once (there is no upper cap — the interval runs from $L$ up to 1). We use $1.645$ because that is the cutoff leaving exactly the 5% we're willing to risk entirely below $L$ .

Verify: $L = 0.8147 > 0.8041$ (the two-sided lower endpoint from Ex 4) — the one-sided floor is indeed higher/tighter, matching the "smaller critical value" forecast ✓. Statement: "We are 95% confident accuracy exceeds 81.5%" ✓.

Cell G — degenerate inputs

Worked example Example 7 — the edges: n=1 and zero spread

Two broken inputs a formula must survive: (a) You have a single observation and unknown $σ$ . (b) All observations are identical, e.g. three runs each scoring exactly $0.90$ .

Forecast: What width should each interval have — zero, finite, or infinite?

Case (a): $n = 1$ , $σ$ unknown.

$s$ needs $n - 1 = 0$ in its denominator → undefined. Degrees of freedom $df = 0$ .
The $t_{0}$ distribution has no finite spread, so the honest CI is $(- \infty, \infty)$ . Why? One point tells you a location but nothing about spread. A single sample cannot estimate its own uncertainty.

Case (b): zero sample variance. $\overset{x}{ˉ} = 0.90$ , and every deviation is 0, so $s = 0$ . $S E = \frac{s}{n} = \frac{0}{3} = 0 \Rightarrow E = t \times 0 = 0, CI = [0.90, 0.90]$ Why a zero-width interval is a red flag, not a triumph: identical values usually mean the metric is quantized or capped (e.g. all runs hit an accuracy ceiling), so the $s = 0$ is an artifact. Prefer Bootstrap Methods here.

Verify (b): $s = (0 + 0 + 0) /2 = 0$ ; $S E = 0$ ; interval collapses to the point ✓. Verify (a): $n - 1 = 0$ makes $s$ formula divide by zero ✓ — the degenerate case is real, not a rounding issue.

Cell H — sample-size planning

Worked example Example 8 — how many samples for ±2%?

You want a 95% CI on a metric with $σ = 0.1$ , and you demand margin $E \leq 0.02$ . Minimum $n$ ?

Forecast: Halving the margin costs how many times more samples — $2 \times$ , $4 \times$ , or $8 \times$ ?

Step 1 — invert the margin formula. From $E = z σ / n$ : $n = (\frac{z σ}{E})^{2}$ Why square? Because $S E \propto 1/ n$ , shrinking the interval by a factor $k$ needs $k^{2}$ times the data. Precision is expensive.

Step 2 — plug in. $n = (\frac{1.96 \times 0.1}{0.02})^{2} = (\frac{0.196}{0.02})^{2} = 9. 8^{2} = 96.04$ Why this step? We substitute the demanded confidence (1.96), the assumed spread (0.1) and the target margin (0.02) into the inverted formula; the result is the exact real-valued sample count that hits $E = 0.02$ dead on.

Step 3 — round up. Always ceil: $n = 97$ . Why round up, never down? $n = 96$ would give a margin slightly larger than 0.02, violating the requirement. Only rounding up guarantees $E \leq 0.02$ .

Verify: At $n = 97$ , $E = 1.96 \times 0.1/ 97 = 0.019899 \leq 0.02$ ✓; at $n = 96$ , $E = 0.020003 > 0.02$ ✗ — so 97 is genuinely the minimum. Forecast answer: halving $E$ quadruples $n$ (the square) ✓.

Cell I — exam twist: raise the confidence

Worked example Example 9 — from 95% to 99%, same data

Reusing Example 1 ( $S E = 0.1$ ), what happens to the interval width when you demand 99% instead of 95% confidence?

Forecast: More confidence with the same data — wider or narrower? By roughly what ratio?

Step 1 — swap the critical value. $z_{0.025} = 1.96 \to z_{0.005} = 2.576$ . Why bigger? To catch $μ$ 99 times out of 100 (instead of 95), the net must extend farther into the tails.

Step 2 — new margin & ratio. $E_{99} = 2.576 \times 0.1 = 0.2576, \frac{E _{99}}{E _{95}} = \frac{0.2576}{0.196} = 1.3143$ Why this step? $S E$ and $n$ are untouched — only the confidence dial moved — so the width scales purely by the ratio of critical values, $2.576/1.96$ . Nothing about the data changed; we simply demanded a bigger net.

Verify: The 99% interval is $\approx 31%$ wider than the 95% one with identical data ✓ — more confidence is bought with width, not with new information. This is the core trade-off behind Bayesian Credible Intervals and Cross-validation reporting.

Recall

Recall Which cell: 400 latency samples, σ unknown?

Cell B — σ unknown but $n \geq 30$ , so $s \approx σ$ and we use $z$ . ::: Use $z$ because large $n$ makes $t_{n - 1} \approx N (0, 1)$ .

Recall Two 95% CIs overlap. Are the estimates significantly different?

Not necessarily — overlap is a weak test. Compute the CI of the difference (Ex 5). ::: If that interval excludes 0, they differ significantly.

Recall Your test accuracy is 10-for-10. Why not report [1.0, 1.0]?

Because $\overset{p}{^} = 1$ makes the normal $S E = 0$ , an artifact. Use the Wilson interval, giving about $[0.722, 1.0]$ . ::: Extreme proportions break the normal approximation.

Recall To halve a margin of error, multiply

$n$ by ___? 4 — because $S E \propto 1/ n$ , so precision costs the square of the improvement factor.