Hypothesis testing and p-values: Exercises — Hinglish deep dive

1.3.20 · D4 · AI-ML › Probability & Statistics › Hypothesis testing and p-values

Shuru karne se pehle, ek picture jo aap poore waqt apne dimaag mein rakhen — woh "extremeness" ka idea jo ek p-value measure karta hai.

Figure — Hypothesis testing and p-values

Har pair ke liye batayein, kaun sa null $H_{0}$ hai aur kaun sa alternative $H_{1}$ : (a) "coin fair hai" vs "coin biased hai"; (b) "naya model 82% se behtar nahi hai" vs "naya model 82% ko beat karta hai".

Recall Solution

Null hamesha boring, no-effect, equality wala statement hota hai. (a) $H_{0} : p = 0.5$ (fair). $H_{1} : p \neq = 0.5$ (biased) — two-sided kyunki "biased" kisi bhi direction mein ja sakta hai. (b) $H_{0} : p = 0.82$ (koi improvement nahi). $H_{1} : p > 0.82$ (better) — one-sided kyunki hum sirf better ki parwah karte hain.

L1.2

Ek test p-value $0.02$ deta hai aur aapne $α = 0.05$ choose kiya. Kya aap $H_{0}$ reject karte hain ya fail to reject? Phir $0.20$ ki p-value ke liye repeat karein.

Recall Solution

Rule: reject karo jab $p < α$ .

$p = 0.02 < 0.05$ → $H_{0}$ reject karein.
$p = 0.20 \geq 0.05$ → $H_{0}$ fail to reject ("accept" nahi).

L1.3

Inme se kaun sa " $p = 0.03$ " ka sahi reading hai? (A) "3% chance hai ki $H_{0}$ true hai." (B) "Agar $H_{0}$ true hota, to itna extreme (ya zyada) data 3% waqt occur karta."

Recall Solution

(B). P-value $P (data ∣ H_{0})$ hota hai, kabhi $P (H_{0} ∣ data)$ nahi. Statement (A) conditioning ko ulta kar deta hai.

Level 2 — Application

L2.1

Aap ek coin $n = 400$ baar flip karte hain aur $X = 220$ heads dekhte hain. $H_{0} : p = 0.5$ vs $H_{1} : p \neq = 0.5$ test karein. Z-statistic compute karein.

Recall Solution

$H_{0}$ ke under, $X \sim Binomial (400, 0.5)$ , isliye $μ = n p = 400 \cdot 0.5 = 200, σ = n p (1 - p) = 400 \cdot 0.5 \cdot 0.5 = 100 = 10.$ Z-statistic mean se standard deviations count karta hai: $Z = \frac{X - μ}{σ} = \frac{220 - 200}{10} = 2.0.$

L2.2

L2.1 continue karte hue, standard normal $P (Z \geq 2) \approx 0.02275$ deta hai. Two-tailed p-value kya hai, aur $α = 0.05$ par aap kya decide karte hain?

Recall Solution

Two-tailed kyunki $H_{1} : p \neq = 0.5$ dono sides par extremeness count karta hai: $p = 2 \cdot P (Z \geq 2) = 2 \cdot 0.02275 = 0.0455.$ Kyunki $0.0455 < 0.05$ , $H_{0}$ reject karein — lekin barely. Neeche ki figure dekhen: hamara $Z = 2$ shaded region ke edge par exactly land karta hai.

L2.3

Ek naya model $n = 150$ samples par $\overset{p}{^} = 0.86$ accuracy score karta hai. Purani accuracy $p_{0} = 0.80$ hai. $H_{0} : p = 0.80$ vs $H_{1} : p > 0.80$ (one-tailed) test karein. $Z$ aur one-tailed p-value nikaalein ( $P (Z \geq 2.06) \approx 0.0197$ ).

Recall Solution

Null ke standard error use karke observed proportion ko standardise karein: $σ_{\overset{p}{^}} = \frac{p _{0} ( 1 - p _{0} )}{n} = \frac{0.80 \cdot 0.20}{150} = \frac{0.16}{150} = 0.0010667 \approx 0.03266.$ $Z = \frac{p ^ - p _{0}}{σ _{\overset{p}{^}}} = \frac{0.86 - 0.80}{0.03266} \approx 1.837.$ One-tailed p-value: $p = P (Z \geq 1.837) \approx 0.0331$ . Kyunki $0.0331 < 0.05$ , $H_{0}$ reject karein — model significantly better hai.

Level 3 — Analysis

L3.1

L2.2 mein aapne $p = 0.0455$ ke saath reject kiya. Suppose aapne pehle se one-tailed test $H_{1} : p > 0.5$ declare kiya hota. P-value kya hoti, aur kya decision badal jaata? Batayein ki ye kya reveal karta hai.

Recall Solution

One-tailed p-value: $p = P (Z \geq 2) = 0.02275$ . Ye two-tailed value se exactly aadhi hai. Decision: reject (abhi bhi $< 0.05$ , ab aur comfortably). Ye kya reveal karta hai: same data sirf ek tail choose karke "zyada significant" ho sakta hai. Isliye aapko one-vs-two-tailed data dekhne se pehle choose karna hota hai — warna aap khud ko threshold ke paas nudge kar sakte hain. Tails switch karne ke baad p-value ko half karna p-hacking ki ek form hai.

L3.2

Do labs same fair coin test karti hain. Lab A ise $n = 100$ baar flip karta hai, Lab B $n = 10, 000$ baar. Dono heads ka proportion $0.55$ observe karte hain. Dono z-statistics compute karein aur comment karein ki kyun "same" result alag conclusions deta hai.

Recall Solution

$σ_{\overset{p}{^}} = 0.5 \cdot 0.5/ n = 0.5/ n$ . Lab A: $σ = 0.5/ 100 = 0.05$ , isliye $Z = (0.55 - 0.5) /0.05 = 1.0$ . Two-tailed $p = 2 P (Z \geq 1) \approx 0.317$ → fail to reject. Lab B: $σ = 0.5/ 10000 = 0.005$ , isliye $Z = (0.55 - 0.5) /0.005 = 10.0$ . $p$ astronomically small → reject. Comment: effect ( $0.55$ vs $0.5$ ) identical hai, lekin bada $n$ standard error ko shrink karta hai, isliye same effect overwhelmingly "significant" ban jaata hai. Statistical significance ka matlab ek bada ya important effect nahi hai — ye sample size ke saath badhta hai (parent ke Mistake 3 mein drug example dekho).

L3.3

Parent note mein die test ne $χ^{2} = 5.8$ diya $df = 5$ ke saath aur $p \approx 0.33$ . $χ^{2}$ recompute karein agar har observed count exactly expected $10$ hota. Resulting p-value interpret karein.

Recall Solution

Har term $(O_{i} - E_{i})^{2} / E_{i} = (10 - 10)^{2} /10 = 0$ , isliye $χ^{2} = 0$ . Ek perfectly-fitting sample jitna ho sake utna un-extreme hota hai, isliye tail probability $P (χ_{5}^{2} \geq 0) = 1$ , yaani $p = 1$ . Interpretation: $p = 1$ prove nahi karta ki die fair hai — iska bas matlab hai ki data fairness ke saath maximally consistent hai. " $H_{0}$ ke saath consistent" kabhi bhi " $H_{0}$ proven" nahi hota.

Level 4 — Synthesis

L4.1

Aap $α = 0.05$ (two-tailed) par z-test run karte hain. Rejection region $∣ Z ∣ \geq z_{crit}$ hai jahan $z_{crit} = 1.96$ hai. (a) Explain karein $1.96$ kaise define hota hai. (b) Dikhayen ki " $H_{0}$ reject karo" equivalent hai "mean ke liye $95%$ confidence interval $μ_{0}$ ko exclude karta hai."

Recall Solution

(a) $z_{crit}$ woh value hai jo har tail mein $α /2 = 0.025$ rakhti hai: $P (Z \geq 1.96) = 0.025$ , isliye dono tails milake exactly $α = 0.05$ hold karti hain. (b) Two-tailed test tab reject karta hai jab $\frac{X ˉ - μ _{0}}{σ / n} \geq 1.96 ⟺ ∣ \overset{ˉ}{X} - μ_{0} ∣ \geq 1.96 \frac{σ}{n} .$ $95%$ confidence interval $\overset{ˉ}{X} \pm 1.96 σ / n$ hai. Ye $μ_{0}$ ko exclude karta hai exactly tab jab $∣ \overset{ˉ}{X} - μ_{0} ∣ > 1.96 σ / n$ — same inequality. Isliye level $α$ par two-sided test aur $(1 - α)$ confidence interval ek hi machine ke do views hain (dekho 1.3.18-Confidence-intervals).

L4.2

Design decision: ek A/B test jahan false positive (ek useless model ship karna) bahut costly hai, lekin false negative (ek achha model miss karna) sasta hai. Kya aap $α$ ko $0.05$ se chhhota ya bada banana chahenge? Type I / Type II errors use karke explain karein.

Recall Solution

$α$ hi Type I error rate (false positive) hai. Kyunki false positives mehenga mistake hai, $α$ kam karein (jaise $0.01$ ). Trade-off: $α$ kam karna $β$ (Type II, false negatives) badhata hai aur power $1 - β$ ghataata hai — lekin yahan false negatives saste hain, isliye ye acceptable hai. Dekho 1.3.21-Type-I-and-Type-II-errors.

L4.3

$H_{0} : p = 0.5$ test karte hue $n = 100$ flips use karke true $p = 0.6$ wale coin ko detect karne ke liye ek one-tailed z-test ( $α = 0.05$ , isliye reject jab $Z \geq 1.645$ ) ki power compute karein. Ye approximation use karein ki $H_{1}$ ke under, $\overset{p}{^} \approx N (0.6, 0.6 \cdot 0.4/100)$ .

Recall Solution

Reject jab $\overset{p}{^} \geq p_{0} + 1.645 p_{0} (1 - p_{0}) / n = 0.5 + 1.645 (0.05) = 0.582$ (null SE $= 0.05$ use karke). Power $= P_{H_{1}} (\overset{p}{^} \geq 0.582)$ . $H_{1}$ ke under, SE $= 0.6 \cdot 0.4/100 = 0.0024 = 0.04899$ . $Z^{'} = \frac{0.582 - 0.6}{0.04899} = \frac{- 0.018}{0.04899} \approx - 0.3675.$ Power $= P (Z^{'} \geq - 0.3675) = P (Z^{'} \leq 0.3675) \approx 0.6434$ . Isliye is test mein genuinely biased ( $p = 0.6$ ) coin ko pakadne ka almost 64% chance hai. Achha nahi — power badhane ke liye aapko bada $n$ chahiye.

Level 5 — Mastery

L5.1

Aap 20 independent hypothesis tests run karte hain, har ek $α = 0.05$ par, aisa data par jahan har $H_{0}$ actually true hai. Kam se kam ek false rejection ki probability kya hai? Kaun sa ek fix ise reduce karta hai?

Recall Solution

Har test mein false positive ka $0.05$ chance hai, isliye no false positive ka $0.95$ chance. Independent, isliye: $P (at least one false positive) = 1 - 0.9 5^{20} = 1 - 0.3585 = 0.6415.$ Almost 64% chance hai ki aap koi aisi cheez "discover" kar lein jo real nahi hai! Ye multiple-comparisons problem hai. Fix: ek multiple-testing correction jaise Bonferroni — har ek ko $α / m = 0.05/20 = 0.0025$ par test karein. Dekho 3.2.12-Multiple-testing-correction.

L5.2

L5.1 mein Bonferroni fix ke baad (per-test level $0.0025$ ), family-wide false-positive probability recompute karein. Kya ye $0.05$ se neeche aayi?

Recall Solution

Per-test survival $= 1 - 0.0025 = 0.9975$ . $P (at least one) = 1 - 0.997 5^{20} = 1 - 0.9512 = 0.0488 < 0.05.$ Haan — Bonferroni family-wide false-positive rate ko $0.05$ ke neeche waapas kheench laata hai, har individual test par kam power ki cost par.

L5.3

Ek drug trial $p = 0.001$ report karta hai lekin average symptom reduction $0.5%$ hai aur $95%$ CI $[0.2%, 0.8%]$ hai. Ek journalist likhta hai "highly significant → ye drug ek breakthrough hai." Resident statistician hone ke naate, do-sentence ka correction likhein.

Recall Solution

"Tiny p-value ka sirf matlab hai ki $0.5%$ effect ke pure chance hone ki bahut kam possibility hai — kaafi bade trial ke saath trivial effects bhi significance tak pahunch jaati hain. Effect size khud ( $0.5%$ , CI $[0.2%, 0.8%]$ ) clinically negligible hai, isliye 'statistically significant' yahan ka matlab 'clinically important' nahi hai — statistical aur practical significance alag cheezein hain (dekho 2.5.7-Statistical-significance-in-experiments)."

Recall One-line self-quiz (reveal each)

P-value two-tailed kab hoti hai? ::: Jab $H_{1}$ " $\neq =$ " ho — kisi bhi direction mein extreme, isliye aap one-tail area ko double karte hain. Reject karein ya nahi agar $p = 0.0455$ , $α = 0.05$ ? ::: Reject karein (barely, kyunki $0.0455 < 0.05$ ). Same effect bade $n$ ke saath "zyada significant" kyun ho jaata hai? ::: Standard error $1/ n$ ki tarah shrink hoti hai, isliye $Z$ badhta hai. Power kya measure karta hai? ::: $1 - β$ , yani false $H_{0}$ ko correctly reject karne ka chance. Family $α = 0.05$ par 20 tests ke liye Bonferroni per-test level? ::: $0.05/20 = 0.0025$ .

Exercises — Hypothesis testing and p-values

Level 1 — Recognition

L1.1

L1.2

L1.3

Level 2 — Application

L2.1

L2.2

L2.3

Level 3 — Analysis

L3.1

L3.2

L3.3

Level 4 — Synthesis

L4.1

L4.2

L4.3

Level 5 — Mastery

L5.1

L5.2

L5.3