Confidence intervals

What Problem Does This Solve?

Point estimates lie. You calculate sample mean $\bar{x} = 170$ cm, but that's just one realization from a random process. The confidence interval quantifies how much we should trust this estimate.

Why do we need this in ML?

Reporting model accuracy: "Test accuracy is 87% ± 2%"
A/B testing: "Treatment increased conversion by 3.2% [2.1%, 4.3%]"
Hyperparameter tuning: quantify uncertainty in cross-validation scores

Derivation from First Principles

Step 1: The Sampling Distribution

Suppose we have a population with true mean $\mu$ (unknown) and variance $\sigma^2$ . We take a sample of size $n$ , compute sample mean:

$\bar{X} = \frac{1}{n} \sum_{i=1}^{n} X_i$

Why this step? $\bar{X}$ is our estimator for $\mu$ . But $\bar{X}$ itself is a random variable because different samples give different values.

By the Central Limit Theorem, for large $n$ :

$\bar{X} \sim N\left(\mu, \frac{\sigma^2}{n}\right)$

Why? Each $X_i$ has mean $\mu$ and variance $\sigma^2$ . The sum has variance $n\sigma^2$ (independent samples). Dividing by $n$ gives variance $\sigma^2/n$ . The CLT says the distribution becomes normal.

Step 2: Standardize to Control Probability

We want to find an interval $[\bar{X} - \text{margin}, \bar{X} + \text{margin}]$ that captures $\mu$ with probability $1-\alpha$ (e.g., 95%).

Standardize $\bar{X}$ :

$Z = \frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \sim N(0, 1)$

Why this step? Standardizing converts our problem to the standard normal, where we know quantiles exactly.

For a $(1-\alpha)$ confidence level, we want:

$P\left(-z_{\alpha/2} \leq \frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \leq z_{\alpha/2}\right) = 1 - \alpha$

where $z_{\alpha/2}$ is the critical value (e.g., $z_{0.025} = 1.96$ for 95%).

Why $\alpha/2$ ? We split the tail probability equally on both sides (two-tailed test).

Step 3: Rearange to Get the Interval

Multiply through by $\sigma/\sqrt{n}$ :

$P\left(-z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} \leq \bar{X} - \mu \leq z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}\right) = 1 - \alpha$

Subtract $\bar{X}$ from all parts:

$P\left(-\bar{X} - z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} \leq -\mu \leq -\bar{X} + z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}\right) = 1 - \alpha$

Multiply by -1 (flips inequalities):

$P\left(\bar{X} - z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} \leq \mu \leq \bar{X} + z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}\right) = 1 - \alpha$

Why this step? We've isolated $\mu$ in the middle. Now the interval depends on observed $\bar{x}$ (lowercase = realized value).

Step 4: When σ is Unknown (The t-Distribution)

In practice, we don't know $\sigma$ . Replace with sample standard deviation $s$ :

$s = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (X_i - \bar{X})^2}$

Why $n-1$ ? Bessel's correction makes $s^2$ an unbiased estimator of $\sigma^2$ (we "used up" one degree of freedom estimating $\mu$ ).

Now the standardized statistic follows a t-distribution with $df = n-1$ degrees of freedom:

$T = \frac{\bar{X} - \mu}{s/\sqrt{n}} \sim t_{n-1}$

Why not normal? Because $s$ is random too. The t-distribution has heavier tails to account for this extra uncertainty. As $n \to \infty$ , $t_{n-1} \to N(0,1)$ .

Worked Examples

Worked example Example 1: ML Model Accuracy

You test a classifier on 100 samples and get 87 correct. Estimate the95% CI for true accuracy. Solution:

Sample proportion: $\hat{p} = 87/100 = 0.87$
For proportions, use normal approximation: $\hat{p} \sim N\left(p, \frac{p(1-p)}{n}\right)$
Standard error: $SE = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} = \sqrt{\frac{0.87 \cdot 0.13}{100}} = 0.0336$ Why this SE? Binomial variance is $np(1-p)$ , so proportion variance is $p(1-p)/n$ .

$\text{CI}_{95\%} = 0.87 \pm 1.96 \cdot 0.0336 = 0.87 \pm 0.066 = [0.804, 0.936]$

Interpretation: We're 95% confident the true accuracy is between 80.4% and 93.6%.

Worked example Example 3: Comparing Two Models

Model A: 82% ± 3% on test set. Model B: 85% ± 4%. Which is better?

Solution: The CIs overlap: A is [79%, 85%], B is [81%, 89%].

Why this matters: We cannot conclude B is definitively better. The difference might be sampling noise. To be rigorous, compute the confidence interval of the difference:

$\text{CI}(\hat{p}_B - \hat{p}_A) = (0.85 - 0.82) \pm \sqrt{SE_A^2 + SE_B^2} \cdot z_{\alpha/2}$

Assuming $SE_A \approx 0.03/1.96 = 0.0153$ , $SE_B \approx 0.0204$ :

$SE_{\text{diff}} = \sqrt{0.0153^2 + 0.0204^2} = 0.0255$

$\text{CI}_{95\%} = 0.03 \pm 1.96 \cdot 0.0255 = 0.03 \pm 0.050 = [-0.020, 0.080]$

Why this step? The CI includes0, so the difference is not statistically significant.

Common Mistakes (Steel-man Your Errors)

How to Use This in Practice

Sample Size Planning

Rearange the margin of error formula to find required $n$ :

$E = z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} \implies n = \left(\frac{z_{\alpha/2} \cdot \sigma}{E}\right)^2$

Why this matters: To get margin of error ±2% with 95% confidence, assuming $\sigma = 0.1$ :

$n = \left(\frac{1.96 \cdot 0.1}{0.02}\right)^2 = 96.04 \approx 97$

80/20 insight: Doubling precision (halving $E$ ) requires 4× the sample size because of the square.

One-sided vs Two-sided

Two-sided: $\mu$ could be higher or lower (most common)
One-sided: only care if $\mu >$ some value (e.g., "is accuracy above 80%?")

For one-sided 95% CI: use $z_{0.05} = 1.645$ instead of $z_{0.025} = 1.96$ .

Recall Explain to a 12-year-old

Imagine you want to know the average height of all students in your school, but you can only measure 30 kids.

You measure them and get 155 cm. But you know if you measured a different 30 kids, you'd get maybe 153 cm or 157 cm. It keeps changing!

A confidence interval is like saying: "I'm pretty sure (95% sure) the real average of the whole school is somewhere between 152 cm and 158 cm."

It's NOT saying "there's a 95% chance the real average is in this range" – the real average is one fixed number, we just don't know it. It's saying "my method of making these ranges is good enough that if I used it 100 times, about 95 of my ranges would catch the real answer."

The more kids you measure, the narrower your range gets, because you're more certain.

Connections Central Limit Theorem – why $\bar{X}$ is approximately normal

Standard Error – why we divide by $\sqrt{n}$
Hypothesis Testing – CIs and p-values are two sides of the same coin
Bootstrap Methods – non-parametric alternative when assumptions fail
Bayesian Credible Intervals – the "95% chance" interpretation you wished you could use
A/B Testing – confidence intervals for treatment effects
Cross-validation – uncertainty in model performance estimates

Active Recall Flashcards

#flashcards/ai-ml

What is a confidence interval?

An interval estimate that, if we repeated the sampling procedure many times, would contain the true population parameter in

(1-\alpha) \cdot 100\%

of cases.

Why is "95% chance the true value is in the interval" wrong?

The true value is fixed (not random). The interval is random. Correct interpretation is about the long-run frequency of the procedure capturing the true value.

Formula for95% CI when σ is known?

\bar{x} \pm 1.96 \cdot \frac{\sigma}{\sqrt{n}}

Why use t-distribution instead of z?

When population standard deviation

\sigma

is unknown and we use sample standard deviation

s

, the extra uncertainty requires the heavier-tailed t-distribution, especially for small samples (

n < 30

What happens to margin of error if you double the sample size?

It decreases by a factor of

\sqrt{2} \approx 1.41

, because the margin of error is proportional to

1/\sqrt{n}

— so multiplying

n

by 2 divides the margin by

\sqrt{2}

How do you find required sample size for a given margin of error E?

n = \left(\frac{z_{\alpha/2} \cdot \sigma}{E}\right)^2

Why do confidence intervals get wider for higher confidence levels?

Higher confidence (e.g., 99% vs 95%) requires larger critical values (

z_{0.005} = 2.576

z_{0.025} = 1.96

), which increases the margin of error.

What are the key assumptions for a valid confidence interval?

1) Random sampling (IID observations), 2) Approximate normality of the sampling distribution (or large

n

for CLT), 3) No extreme outliers or violations of the model.

If two95% CIs overlap, what can you conclude?

You cannot conclude the parameters are equal. You need to compute the confidence interval for the difference to test for significance.

What is Bessel's correction and why use n-1?

Using

n-1

instead of

n

in sample variance makes

s^2

an unbiased estimator of

\sigma^2

, accounting for one degree of freedom lost in estimating the mean.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, confidence interval ka core idea ye samajhna zaroori hai. Jab tum kisi population ka average nikalna chahte ho, jaise ki 100 logon ki average height, toh tumhe ek single number milta hai jaise 170 cm. Lekin problem ye hai ki agar tum dobara koi aur 100 log sample karo, toh ye number thoda change ho jayega. Matlab tumhara point estimate (170 cm) hamesha thoda "jhoot" bolta hai, kyunki wo bas ek random sample ka result hai. Confidence interval isko fix karta hai — wo tumhe ek range deta hai, jaise 167 se 173 cm, aur kehta hai "main 95% confident hoon ki asli population average is range mein hai." Ek important baat: iska matlab ye NAHI hai ki true value ke 95% chance hai is range mein hone ke — true value toh fixed hai. Iska matlab ye hai ki agar tum ye poora sampling procedure 100 baar repeat karo, toh lagbhag 95 baar tumhara interval sahi value ko capture kar lega.

Ab thodi si maths samajh lo. Central Limit Theorem kehta hai ki sample mean $\bar{X}$ approximately normal distribution follow karta hai, jiska mean $\mu$ hai aur variance $\sigma^2/n$ . Isko standardize karke hum Z-score bana lete hain, aur phir jo critical value hoti hai (jaise 95% ke liye 1.96), usse multiply karke margin of error nikalte hain. Formula simple hai: $\bar{x} \pm z_{\alpha/2} \cdot \sigma/\sqrt{n}$ . Notice karo ki jitna bada $n$ hoga, utna chota interval hoga — matlab zyada data se zyada confidence. Aur jab humein population ki actual $\sigma$ nahi pata (jo real life mein zyadatar hota hai), tab hum sample se estimate kiya hua $s$ use karte hain, aur normal ki jagah t-distribution use karte hain jiske tails thode heavy hote hain — kyunki $s$ khud bhi ek random quantity hai, toh extra uncertainty ka dhyan rakhna padta hai.

Ye cheez ML mein kyun matter karti hai? Kyunki jab tum kisi model ki accuracy report karte ho jaise "87%", toh wo bhi bas ek estimate hai. Behtar hai kehna "87% ± 2%" — isse pata chalta hai ki tumhara result kitna reliable hai. A/B testing mein bhi, agar tum bolo "conversion 3.2% badh gaya", toh confidence interval [2.1%, 4.3%] batata hai ki ye result genuine hai ya bas random noise. Aur hyperparameter tuning mein cross-validation scores ki uncertainty samajhne ke liye bhi yahi concept kaam aata hai. Basically, confidence interval tumhe honest banata hai — bas ek number pe blindly bharosa karne ke bajaye tum uncertainty ko bhi acknowledge karte ho, jo ki proper data science ka foundation hai.

What Problem Does This Solve?

Derivation from First Principles

Step 1: The Sampling Distribution

Step 2: Standardize to Control Probability

Step 3: Rearange to Get the Interval

Step 4: When σ is Unknown (The t-Distribution)

Worked Examples

Common Mistakes (Steel-man Your Errors)

How to Use This in Practice

Sample Size Planning

One-sided vs Two-sided

Connections Central Limit Theorem – why $\bar{X}$ is approximately normal

Active Recall Flashcards

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability & Statistics

Connections

Connections Central Limit Theorem – why Xˉ\bar{X}Xˉ is approximately normal

Go deeper — visual, from zero

Test yourself — Probability & Statistics

Connections

Connections Central Limit Theorem – why $\bar{X}$ is approximately normal