1.3.21 · D3 · AI-ML › Probability & Statistics › Confidence intervals
Yeh Confidence intervals ka ek worked-examples deep dive hai. Parent note ne formulas build kiye the; yahaan hum unhe har tarah ke inputs ke khilaf hammer karte hain jo ek confidence-interval problem mein aa sakti hain — bade samples, chhote samples, proportions, proportions ka difference, means ka difference, one-sided questions, aur woh degenerate edges jahan formula almost break karta hai.
Intuition Woh ek picture jo hamesha dimag mein rakhni chahiye
Ek confidence interval ek net hai jiska half-width E hai (yeh "margin of error" hai) jo aapke sample estimate ke around centered hai. Alag-alag problems sirf teen dials badlati hain: spread ke per unit net kitna choda hai (z ya t ), raw spread kitna hai (σ ya s ), aur sample size ke saath spread kitna shrink hota hai (n ).
net half-width E = how confident? ( z or t ) × standard error n spread
Neeche ka har example sirf un teen dials ke liye values choose kar raha hai. Prerequisites Central Limit Theorem aur Standard Error mein hain.
Definition Do words jo hum har jagah use karenge
Margin of error E = interval ki half-width . Poora interval x ˉ − E se x ˉ + E tak hai, isliye uski total length 2 E hai. Half-width ko full width se kabhi confuse mat karo.
Degrees of freedom (df ) = independent information ke kitne pieces bache hain spread estimate karne ke liye, jab aap pehle se mean estimate karne mein kuch spend kar chuke hain. Ek sample of size n ke liye aap x ˉ par 1 spend karte hain, bachhta hai df = n − 1 .
Figure s01 (pehle yeh padhein): black curve sampling distribution of the mean hai; red bar woh confidence-interval "net" hai jiska half-width E hai, jo observed x ˉ (black dot) ke around throw kiya gaya hai. True μ (black cross) fixed hai — net woh hai jo sample se sample tak move karta hai. Neeche ka har example bas is red bar ko resize karta hai.
Kuch bhi work karne se pehle, aao har case class enumerate karte hain jo ek CI problem mein ho sakti hai. Ek dense grid ki jagah, neeche ka decision tree padhein — usme wahi content visual-first form mein hai, aur har leaf us example ka naam bataati hai jo use fill karta hai.
Figure s02 (is poore page ka map): apne case par land karne ke liye arrows follow karein. Red leaf ("small n ⇒ use t ") woh hai jise beginners aksar galat karte hain. Har leaf apne example number ke saath tagged hai.
Cell D - binomial SE - Ex 4
Cell D-edge - Wilson - Ex 4b
Cell E - add variances - Ex 5
Cell E2 - add variances - Ex 5b
Cell F - all alpha one tail - Ex 6
n equals 1 or zero spread
Cell G - degenerate - Ex 7
Har cell ke liye dials:
Confidence dial: z 0.025 = 1.96 (95%), z 0.005 = 2.576 (99%), z 0.05 = 1.645 (90% two-sided ya 95% one-sided).
Spread dial: σ agar known ho, warna s (sample std, Confidence intervals wala n − 1 ke saath).
Shrink dial: n .
Worked example Example 1 — sensor calibration
Ek temperature sensor factory se known hai ki uska measurement noise standard deviation σ = 0. 5 ∘ C hai. Aap n = 25 readings lete hain aur x ˉ = 21. 0 ∘ C paate hain. True temperature ke liye 95% CI banao.
Forecast: 25 readings aur ek chhote known noise ke saath, kya aap expect karte hain ki interval roughly ± 0.02 , ± 0.2 , ya ± 2 degrees hoga? Aage padhne se pehle guess karo.
Step 1 — tool pick karo. σ known hai, isliye t nahi, z use karo.
Yeh step kyun? t -distribution sirf isliye exist karti hai taaki σ ke baare mein hamari uncertainty ka hisaab laga sake. Yahaan woh hai nahi, isliye z exact hai.
Step 2 — standard error.
S E = n σ = 25 0.5 = 5 0.5 = 0.1
Yeh step kyun? S E sample mean ka spread hai, ek reading ka nahi. 25 values ko average karne se mean ek single reading se 25 = 5 × tighter ho jaata hai.
Step 3 — margin of error.
E = z 0.025 ⋅ S E = 1.96 × 0.1 = 0.196
Yeh step kyun? z 0.025 = 1.96 standard normal ka woh point hai jo har tail mein 2.5% chhod deta hai (total 5% = woh α jo hum miss hone de sakte hain).
Step 4 — assemble karo. Interval x ˉ minus E left par aur x ˉ plus E right par hai:
CI 95% = 21.0 ± 0.196 = [ 21.0 − 0.196 , 21.0 + 0.196 ] = [ 20.804 , 21.196 ]
Yeh step kyun? E half-width hai; hum ise point estimate ke left mein ek baar aur right mein ek baar lay off karte hain, total length 2 E = 0.392 dete hain.
Verify: Half-width 0.196 0.02 aur 2 ke beech hai — middle forecast se match karta hai. Full length = 2 × 0.196 = 0.392 , aur 21.196 − 20.804 = 0.392 ✓. Units poori tarah °C hain ✓. Sanity: 1.96 SEs ek baal 2 SEs se kam hai, isliye ≈ 2 × 0.1 = 0.2 ✓.
Worked example Example 2 — model latency, big log
Aap ek model ki inference latency n = 400 requests par log karte hain. Sample mean x ˉ = 52 ms, sample std s = 10 ms. True mean latency ke liye 95% CI?
Forecast: 400 samples ke saath, kya aap t ya z use karoge? Kya interval Example 1 ke ± 0.196 se wider ya narrower hoga?
Step 1 — tool choice. σ unknown, lekin n = 400 ≥ 30 , isliye s ≈ σ aur hum z use kar sakte hain.
Yeh step kyun? Large n ke liye, t 399 visually N ( 0 , 1 ) se practically alag nahi dikhta (heavier tails gayab ho jaati hain). z use karne mein almost kuch nahi jata.
Step 2 — standard error.
S E = n s = 400 10 = 20 10 = 0.5 ms
Yeh step kyun? Hum unknown σ ki jagah sample std s le lete hain aur, bilkul Example 1 ki tarah, n se divide karte hain taaki ek single request ki jagah mean ka spread mile — 400 requests mean ko ek request se 400 = 20 × tighter banate hain.
Step 3 — margin & interval.
E = 1.96 × 0.5 = 0.98 , CI 95% = 52 ± 0.98 = [ 51.02 , 52.98 ] ms
Yeh step kyun? Cell A jaisa hi recipe — hum 95% critical value 1.96 ko standard error S E = 0.5 se multiply karte hain taaki "spread of the mean" ko half-width mein convert karein, phir woh half-width x ˉ = 52 ke left aur right mein lay off karein. 1.96 (not, say, 1) se multiply karna hi coverage ko exactly 95% par pin karta hai; x ˉ par centering hi interval ko unknown μ ke baare mein honest statement banata hai.
Verify: t 0.025 , 399 ≈ 1.966 — 1.96 ko 1.966 se replace karne par E 0.980 se 0.983 ho jaata hai, ek 0.3% difference. Negligible, "use z " decision confirm karta hai ✓. Units ms ✓.
Worked example Example 3 — paanch A/B experiment lifts
Ek cautious team ne same experiment n = 5 baar run kiya, conversion lifts (% mein) mile: 2.0 , 3.5 , 1.0 , 4.0 , 2.5 . True lift ke liye 90% CI banao.
Forecast: Sirf 5 points ke saath, kya t , z = 1.645 se kaafi bada hoga? Guess karo ki interval zero span karta hai ya nahi.
Step 1 — sample mean.
x ˉ = 5 2.0 + 3.5 + 1.0 + 4.0 + 2.5 = 5 13.0 = 2.6
Step 2 — sample std Bessel ke n − 1 ke saath. 2.6 se deviations: − 0.6 , 0.9 , − 1.6 , 1.4 , − 0.1 . Squares: 0.36 , 0.81 , 2.56 , 1.96 , 0.01 , summing to 5.70 .
s = 5 − 1 5.70 = 1.425 = 1.1937
n − 1 kyun? Humne pehle se ek degree of freedom (upar wala [!definition] dekhein) x ˉ pin karne mein spend kiya; df = 4 (5 nahi) se divide karna s 2 ko un-bias karta hai. Confidence intervals dekhein.
Step 3 — standard error.
S E = 5 s = 2.2361 1.1937 = 0.5338
Step 4 — critical value. df = n − 1 = 4 ke saath aur 90% two-sided ⇒ t 0.05 , 4 = 2.132 .
t kyun aur itna bada kyun? n < 30 aur σ unknown. Sirf 4 degrees of freedom ke saath tails fat hain, isliye 2.132 ≫ 1.645 — interval ko honest rehne ke liye wider hona chahiye.
Step 5 — interval.
E = 2.132 × 0.5338 = 1.138 , CI 90% = 2.6 ± 1.138 = [ 1.462 , 3.738 ]
Verify: Interval 0 include nahi karta ⇒ lift 90% par significant hai. Compare karein: agar hum galti se z = 1.645 use karte, E = 0.878 , [ 1.72 , 3.48 ] milta — falsely narrower . t correction ne ise ≈ 30% widened kiya ✓.
Figure s03 (kyun small n ko t chahiye): black curve standard normal hai; red curve df = 4 ke saath t -distribution hai. Red curve ki fatter tails notice karein — isliye uska 90% cutoff (red dashed, 2.132 par) normal ke cutoff (black dashed, 1.96 par) se aage baithta hai. Fatter tails = wider net = tiny samples ke liye honest extra caution. Yeh Example 3, Step 4 ke peeche ki picture hai.
Worked example Example 4 — classifier accuracy
Ek classifier 100 test cases mein se 87 par sahi hai. True accuracy ke liye 95% CI?
Forecast: Kya half-width ± 0.03 , ± 0.07 , ya ± 0.15 ke kareeb hogi?
Step 1 — point estimate. p ^ = 87/100 = 0.87 .
Step 2 — proportion standard error.
S E = n p ^ ( 1 − p ^ ) = 100 0.87 × 0.13 = 0.0011310 = 0.033630
Yeh SE kyun? Ek "correct/incorrect" count binomial hai jiska variance n p ( 1 − p ) hai; ek proportion n se do baar divide karta hai, variance p ( 1 − p ) / n deta hai. Standard Error dekhein.
Step 3 — interval.
E = 1.96 × 0.033630 = 0.065915 , CI 95% = 0.87 ± 0.0659 = [ 0.8041 , 0.9359 ]
Yeh step kyun? Mean cases jaisa bilkul wahi recipe — sirf yeh badla ki humne kaun sa S E formula feed kiya. 1.96 (95%) aur "half-width dono taraf lay off karo" logic unchanged hai.
Verify: Half-width 0.066 ≈ ± 0.07 forecast ✓. Dono endpoints [ 0 , 1 ] mein hain (valid probabilities) ✓. Rule-of-thumb check n p = 87 ≥ 5 aur n ( 1 − p ) = 13 ≥ 5 , isliye normal approximation legitimate hai ✓.
Worked example Example 4b — extreme-proportion edge (Wilson / Jeffreys)
Ek model sab 10 mein se 10 test cases par sahi hai: p ^ = 10/10 = 1.0 . 95% CI banao.
Forecast: Naïve formula S E = 1 ⋅ 0/10 = 0 deta hai, toh yeh claim karta hai interval [ 1 , 1 ] hai — "hum certain hain accuracy exactly 100% hai". Kya 10-for-10 sach mein perfection prove karta hai?
Step 1 — spot karo ki normal formula kyun collapse karti hai. Jab p ^ = 0 ya p ^ = 1 , p ^ ( 1 − p ^ ) = 0 , isliye S E = 0 aur interval ek point par degenerate ho jaata hai.
Yeh step kyun? Normal approximation ko n p ^ ≥ 5 aur n ( 1 − p ^ ) ≥ 5 chahiye; yahaan n ( 1 − p ^ ) = 0 hai, isliye approximation simply invalid hai, exact nahi.
Step 2 — Wilson interval use karo. Wilson score interval boundary par kabhi zero width nahi deta aur hamesha [ 0 , 1 ] ke andar rehta hai:
center = 1 + n z 2 p ^ + 2 n z 2 , half-width = 1 + n z 2 z n p ^ ( 1 − p ^ ) + 4 n 2 z 2
p ^ = 1 , n = 10 , z = 1.96 (toh z 2 = 3.8416 ) ke saath: center = 1 + 3.8416/10 1 + 3.8416/20 = 1.38416 1.19208 = 0.8613 , aur half-width = 1.38416 1.96 0 + 400 3.8416 = 1.41603 × 0.098 = 0.1388 , [ 0.7224 , 0.9999 ] deta hai.
Yeh step kyun? Wilson z 2 /2 imaginary successes aur z 2 /2 failures add karta hai, estimate ko boundary se nudge karta hai taaki uncertainty honestly report ho.
Verify: Wilson lower bound 0.7224 < 1 — yeh theek se 10 trials se certainty claim karne se mana karta hai ✓; upper bound 0.9999 ≤ 1 (100% se exceed nahi ho sakta) ✓. (Jeffreys interval, Beta prior par based, edges par similarly behave karta hai — ise Bayesian cousin ke roop mein mention karein, Bayesian Credible Intervals dekhein.)
Worked example Example 5 — kya Model B sach mein better hai?
Model A: 82% correct, reported margin ± 3% (95%). Model B: 85% correct, margin ± 4% . Kya B significantly better hai?
Forecast: Unke intervals overlap karte hain. Kya overlap alone "koi difference nahi" prove karta hai? (Trap!)
Step 1 — har SE ko uske margin se recover karo. Ek 95% margin 1.96 × S E hai, isliye S E = E /1.96 .
S E A = 1.96 0.03 = 0.015306 , S E B = 1.96 0.04 = 0.020408
Yeh step kyun? Problem humein margins deta hai, standard errors nahi. Kyunki ek margin SE se 1.96 multiply karke banta hai, hum us construction ko ulta chalane ke liye 1.96 se divide karte hain aur har model ki raw uncertainty recover karte hain.
Step 2 — difference ka SE (variances add karo). Independent estimates ke liye, variances add hote hain:
S E diff = S E A 2 + S E B 2 = 0.01530 6 2 + 0.02040 8 2 = 0.00065598 = 0.025612
Variances kyun add karein? Ek difference mein uncertainty do independent uncertainties ka Pythagorean sum hai, kabhi plain sum nahi. (Yeh difference ke liye ek variance-sum SE hai — ise "pooled SE" mat bolo, jo ek alag construction hai jo common variance assume karta hai.) A/B Testing dekhein.
Step 3 — difference ka CI. p ^ B − p ^ A = 0.85 − 0.82 = 0.03 .
E = 1.96 × 0.025612 = 0.050200 , CI 95% = 0.03 ± 0.0502 = [ − 0.0202 , 0.0802 ]
Yeh step kyun? Phir se wahi universal recipe: point estimate (yahaan difference 0.03 ) ± (critical value × difference ka SE). Hum difference par center karte hain kyunki wahi woh quantity hai jo question poochh raha hai.
Verify: Interval 0 contain karta hai , isliye difference significant nahi — B ka point estimate higher hone ke bawajood. Lesson: individual CIs ka overlap karna ek weak test hai; difference CI correct hai, aur yahaan yeh "not conclusive" confirm karta hai ✓. Related: Hypothesis Testing .
Worked example Example 5b — kya naye optimizer ne loss kum kiya?
Old optimizer: n 1 = 36 runs, mean loss x ˉ 1 = 0.50 , sample std s 1 = 0.12 . New optimizer: n 2 = 36 runs, mean loss x ˉ 2 = 0.44 , sample std s 2 = 0.09 . Mean loss μ 1 − μ 2 mein drop ke liye 95% CI?
Forecast: Point drop 0.06 hai. Kya CI 0 clear karega (real improvement) ya straddle karega?
Step 1 — har mean ka SE. Large n (≥ 30 ), σ unknown ⇒ Cell B logic, s ke saath z use karo.
S E 1 = 36 0.12 = 0.02 , S E 2 = 36 0.09 = 0.015
Yeh step kyun? Har mean ka apna standard error hota hai, bilkul Example 2 ki tarah compute kiya gaya.
Step 2 — difference ka SE (variances add karo).
S E diff = S E 1 2 + S E 2 2 = 0.0 2 2 + 0.01 5 2 = 0.000625 = 0.025
Ex 5 jaisa hi rule kyun? Do independent means Pythagorean-sum uncertainty ke saath differ karte hain, spirit mein two-proportion case se identical — bas feed kiya gaya SE formula means se aaya, proportions se nahi.
Step 3 — difference ka CI.
E = 1.96 × 0.025 = 0.049 , CI 95% = 0.06 ± 0.049 = [ 0.011 , 0.109 ]
Yeh step kyun? Observed drop x ˉ 1 − x ˉ 2 = 0.06 par center karo; half-width 0.049 dono taraf lay off karo.
Verify: Interval [ 0.011 , 0.109 ] 0 exclude karta hai , isliye loss drop 95% par significant hai ✓ — Example 5 se unlike, yeh difference zero clear karta hai. Units poori tarah loss units hain ✓.
Worked example Example 6 — accuracy par ek floor
Same classifier Ex 4 se (p ^ = 0.87 , S E = 0.033630 , n = 100 ). Aapka deployment gate sirf yeh poochhhta hai: "Kya hum 95% confident hain ki accuracy kisi floor L se zyada hai?" L dhundho.
Forecast: Kya one-sided critical value 1.96 se bada ya chhota hona chahiye?
Step 1 — saara α ek tail mein daalo. Ek one-sided 95% lower bound ke liye, saara 5% left tail mein jaata hai, isliye critical value z 0.05 = 1.645 hai, 1.96 nahi .
Kyun chhota? Hum sirf ek side par galat hone se bachte hain, isliye hum ek less extreme cutoff afford kar sakte hain — one-sided bound ek two-sided interval ke har side se tighter hai.
Step 2 — lower confidence bound.
L = p ^ − 1.645 × S E = 0.87 − 1.645 × 0.033630 = 0.87 − 0.055322 = 0.814678
Yeh step kyun? Ek one-sided lower bound sirf ek baar half-width subtract karta hai (koi upper cap nahi — interval L se 1 tak jaata hai). Hum 1.645 use karte hain kyunki woh woh cutoff hai jo exactly 5% chhod deta hai jo hum risk karne ko taiyaar hain entirely L ke neeche.
Verify: L = 0.8147 > 0.8041 (Ex 4 se two-sided lower endpoint) — one-sided floor indeed higher/tighter hai, "smaller critical value" forecast se match karta hai ✓. Statement: "Hum 95% confident hain ki accuracy 81.5% se zyada hai" ✓.
Worked example Example 7 — edges: n=1 aur zero spread
Do broken inputs jo formula ko survive karne chahiye:
(a) Aapke paas ek single observation hai aur σ unknown hai.
(b) Saare observations identical hain, e.g. teen runs har ek exactly 0.90 score karte hain.
Forecast: Har interval ki width kitni honi chahiye — zero, finite, ya infinite?
Case (a): n = 1 , σ unknown.
s ko n − 1 = 0 apne denominator mein chahiye → undefined . Degrees of freedom df = 0 .
t 0 distribution ki koi finite spread nahi hai, isliye honest CI ( − ∞ , ∞ ) hai.
Kyun? Ek point location bata sakta hai lekin spread ke baare mein kuch nahi . Ek single sample apni khud ki uncertainty estimate nahi kar sakta.
Case (b): zero sample variance. x ˉ = 0.90 , aur har deviation 0 hai, isliye s = 0 .
S E = n s = 3 0 = 0 ⇒ E = t × 0 = 0 , CI = [ 0.90 , 0.90 ]
Kyun ek zero-width interval ek triumph nahi, ek red flag hai: identical values usually matlab hai metric quantized ya capped hai (e.g. saare runs ek accuracy ceiling hit karte hain), isliye s = 0 ek artifact hai. Yahaan Bootstrap Methods prefer karein.
Verify (b): s = ( 0 + 0 + 0 ) /2 = 0 ; S E = 0 ; interval point par collapse ✓. Verify (a): n − 1 = 0 s formula ko zero se divide karwata hai ✓ — degenerate case real hai, rounding issue nahi.
Worked example Example 8 — ±2% ke liye kitne samples?
Aap σ = 0.1 wale ek metric par 95% CI chahte hain, aur aap margin E ≤ 0.02 demand karte hain. Minimum n ?
Forecast: Margin ko half karne ke liye kitne zyada samples chahiye — 2 × , 4 × , ya 8 × ?
Step 1 — margin formula invert karo. E = z σ / n se:
n = ( E z σ ) 2
Square kyun? Kyunki S E ∝ 1/ n hai, interval ko k factor se shrink karne ke liye k 2 zyada data chahiye. Precision mehngi hai.
Step 2 — plug in karo.
n = ( 0.02 1.96 × 0.1 ) 2 = ( 0.02 0.196 ) 2 = 9. 8 2 = 96.04
Yeh step kyun? Hum demanded confidence (1.96), assumed spread (0.1) aur target margin (0.02) ko inverted formula mein substitute karte hain; result woh exact real-valued sample count hai jo E = 0.02 exactly hit karta hai.
Step 3 — round up karo. Hamesha ceil karo: n = 97 .
Round up kyun, neeche nahi? n = 96 ek margin deta jo 0.02 se thoda bada hota, requirement violate karta. Sirf rounding up E ≤ 0.02 guarantee karta hai.
Verify: n = 97 par, E = 1.96 × 0.1/ 97 = 0.019899 ≤ 0.02 ✓; n = 96 par, E = 0.020003 > 0.02 ✗ — isliye 97 genuinely minimum hai. Forecast answer: E half karne par n quadruple hota hai (square) ✓.
Worked example Example 9 — 95% se 99% tak, same data
Example 1 reuse karte hue (S E = 0.1 ), kya hota hai interval width ko jab aap same data ke saath 99% confidence demand karte hain?
Forecast: Same data ke saath zyada confidence — wider ya narrower? Roughly kis ratio se?
Step 1 — critical value swap karo. z 0.025 = 1.96 → z 0.005 = 2.576 .
Bada kyun? μ ko 100 mein se 99 baar pakadne ke liye (95 ki jagah), net ko tails mein aur aage extend karna hoga.
Step 2 — naya margin & ratio.
E 99 = 2.576 × 0.1 = 0.2576 , E 95 E 99 = 0.196 0.2576 = 1.3143
Yeh step kyun? S E aur n untouched hain — sirf confidence dial move hua — isliye width purely critical values ke ratio se scale hoti hai, 2.576/1.96 . Data ke baare mein kuch nahi badla; humne sirf ek bada net maanga.
Verify: 99% interval 95% wale se identical data ke saath ≈ 31% wider hai ✓ — zyada confidence width se kharidi jaati hai, naye information se nahi. Yeh woh core trade-off hai jo Bayesian Credible Intervals aur Cross-validation reporting ke peeche hai.
Recall Kaun sa cell: 400 latency samples, σ unknown?
Cell B — σ unknown lekin n ≥ 30 , isliye s ≈ σ aur hum z use karte hain. ::: z use karo kyunki large n t n − 1 ≈ N ( 0 , 1 ) banata hai.
Recall Do 95% CIs overlap karte hain. Kya estimates significantly different hain?
Not necessarily — overlap ek weak test hai. Difference ka CI compute karo (Ex 5). ::: Agar woh interval 0 exclude karta hai, toh woh significantly differ karte hain.
Recall Aapki test accuracy 10-for-10 hai. [1.0, 1.0] kyun report nahi karte?
Kyunki p ^ = 1 normal S E = 0 banata hai, jo ek artifact hai. Wilson interval use karo, roughly [ 0.722 , 1.0 ] deta hai. ::: Extreme proportions normal approximation tod dete hain.
Recall Margin of error half karne ke liye,
n ko ___ se multiply karo?
4 — kyunki S E ∝ 1/ n , isliye precision improvement factor ka square khareedti hai.
Mnemonic z vs t vs bootstrap vs Wilson choose karna
σ known → z. σ unknown & big n → z. σ unknown & small n → t. Extreme/tiny proportion → Wilson. Skewed/capped data → Bootstrap Methods .