Hypothesis testing and p-values

1.3.20 · AI-ML › Probability & Statistics

Hypothesis Testing Kya Hai?

Hypothesis testing ek statistical framework hai jo sample data use karke kisi population ke baare mein do competing claims mein se ek decide karta hai.

Process (first principles se):

Maano $H_{0}$ sach hai (jab tak prove na ho, innocent maano).
Apne data se ek test statistic compute karo—ek number jo summarize kare ki tera observation $H_{0}$ ki prediction se kitna door hai.
p-value calculate karo: $P (test statistic as extreme as observed ∣ H_{0} true)$ .
Decision rule: Agar $p < α$ (significance level, aksar 0.05), toh $H_{0}$ reject karo. Warna, reject karne mein fail ho jao.

p-value Derive Karna (Coin Example)

Setup: Tum ek coin $n = 100$ baar flip karte ho, observe karte ho $X = 65$ heads. Test karo $H_{0} : p = 0.5$ vs $H_{1} : p \neq = 0.5$ .

Step 1: Test Statistic Choose Karo

$H_{0}$ ke under, $X \sim Binomial (n = 100, p = 0.5)$ . Bade $n$ ke liye, normal approximation use karo:

Z = \frac{X - μ}{σ} = \frac{X - n p}{n p ( 1 - p )}

Kyun? CLT kehta hai sample proportions approximately normal hote hain. $Z$ measure karta hai "mean se kitne standard deviations door."

Derivation:

Mean: $μ = n p = 100 \cdot 0.5 = 50$
Std dev: $σ = n p (1 - p) = 100 \cdot 0.5 \cdot 0.5 = 5$
Observed: $Z = \frac{65 - 50}{5} = 3$

Step 2: p-value Compute Karo

Kyunki $H_{1} : p \neq = 0.5$ two-tailed hai (hum bias ki kisi bhi direction mein care karte hain), p-value hai:

p -value = P (∣ Z ∣ \geq 3 ∣ H_{0}) = 2 \cdot P (Z \geq 3)

2 kyun? Hum dono tails count karte hain: $Z \geq 3$ (bahut zyada heads) aur $Z \leq - 3$ (bahut zyada tails).

Standard normal table se: $P (Z \geq 3) \approx 0.0013$

p -value = 2 \cdot 0.0013 = 0.0026

Step 3: Decision

$p = 0.0026 < 0.05 = α$ → $H_{0}$ reject karo. Conclusion: Strong evidence hai ki coin biased hai.

p-value ka matlab: Agar coin fair hota, toh sirf 0.26% chance hota ki hum itna extreme result dekhte (ya aur zyada extreme). Yeh itna rare hai ki "fair coin" assumption par doubt hota hai.

Worked Examples

Worked example Example 1: Ek Model ka A/B Testing

Scenario: Tumhare purane model ki accuracy 82% hai. Ek naya model 200 test samples par 87% laata hai. Kya yeh improvement real hai ya sirf luck?

Setup:

$H_{0}$ : Naye model ki accuracy = 82% (koi improvement nahi)
$H_{1}$ : Naye model ki accuracy > 82% (one-tailed, hum sirf care karte hain agar yeh better ho)
Observed: $\overset{p}{^} = 0.87$ , $n = 200$ Step 1: Test Statistic $H_{0}$ ke under, accuracy $Binomial (200, 0.82)$ follow karta hai. Normal approximation: $Z = \frac{p ^ - p _{0}}{p _{0} ( 1 - p _{0} ) / n} = \frac{0.87 - 0.82}{0.82 \cdot 0.18/200}$

Yeh step kyun? Hum difference ko standardize kar rahe hain taaki measure kar sakein yeh kitna unusual hai.

$Z = \frac{0.05}{0.1476/200} = \frac{0.05}{0.0272} \approx 1.84$

Step 2: p-value (one-tailed) $p = P (Z \geq 1.84) \approx 0.033$

One-tailed kyun? Hum care nahi karte agar naya model worse ho; hum sirf reject karte hain agar yeh better ho.

Step 3: Decision $p = 0.033 < 0.05$ → $H_{0}$ reject karo. Improvement statistically significant hai.

Worked example Example 2: Dice Fair Hai Ya Nahi, Test Karo

Tum ek die 60 baar roll karte ho, yeh counts milte hain:

Face	1	2	3	4	5	6
Count	5	8	12	15	10	10

Sawaal: Kya die fair hai?

Setup: $H_{0}$ : Sabhi faces equally likely hain (har ek $p = 1/6$ ). Ek chi-squared test use karo.

$χ^{2} = \sum_{i = 1}^{6} \frac{( O _{i} - E _{i} ) ^{2}}{E _{i}}$

Yeh formula kyun? Yeh measure karta hai ki observed counts ( $O_{i}$ ) $H_{0}$ ke under expected counts ( $E_{i} = 60/6 = 10$ ) se kitna door hain.

Calculation: $χ^{2} = \frac{( 5 - 10 ) ^{2}}{10} + \frac{( 8 - 10 ) ^{2}}{10} + \frac{( 12 - 10 ) ^{2}}{10} + \frac{( 15 - 10 ) ^{2}}{10} + \frac{( 10 - 10 ) ^{2}}{10} + \frac{( 10 - 10 ) ^{2}}{10}$ $= \frac{25 + 4 + 4 + 25 + 0 + 0}{10} = 5.8$

Yeh step kyun? Har term ko square kiya jaata hai taaki dono directions ke deviations penalty mile, aur expected se divide karke standardize kiya jaata hai.

Degrees of freedom: $df = 6 - 1 = 5$ (Kyun? Ek constraint: total 60 hona chahiye). Chi-squared table se: $P (χ_{5}^{2} \geq 5.8) \approx 0.33$

Decision: $p = 0.33 > 0.05$ → $H_{0}$ reject karne mein fail. Itna evidence nahi hai ki kahein die unfair hai.

Common Mistakes

The Significance Level $α$

$α$ kya hai? Significance level woh threshold hai jiske liye "itna weird ki reject kar do." Aam tor par 0.05, lekin yeh arbitrary hai.

0.05 kahan se aaya? Historical convention (R.A. Fisher, 1920s). Yeh do errors ke beech balance karta hai:

Type I error ( $α$ ): $H_{0}$ ko reject karna jab yeh sach ho (false positive)
Type II error ( $β$ ): $H_{0}$ ko reject karne mein fail hona jab yeh galat ho (false negative)

Power = 1 - β = P (reject H_{0} ∣ H_{1} true)

Trade-off: Chhota $α$ → kam false positives, lekin zyada false negatives (less power).

Active Recall Practice

Recall Feynman Explanation (Ek 12-saal ke bachche ko samjhao)

Theek hai, socho tumhara dost kehta hai, "Mere paas ek magic coin hai jo hamesha heads pe aata hai!" Tum skeptical ho, toh tum kehte ho, "Prove karo." Woh use 10 baar flip karta hai, aur yeh 7 baar heads pe aata hai.

Ab, sawaal yeh hai: Kya coin sach mein magic hai, ya unhone bas luck paaya?

Hypothesis testing detective banne jaisa hai. Tum shuru karte ho yeh maanke ki coin normal hai ("null hypothesis"). Phir tum poochho, "Agar coin normal hota, toh kitni baar main 10 mein se 7 ya zyada heads dekhta?" Tum chance calculate karte ho—pata chalta hai yeh kareeban 17% hai (roughly 6 mein se 1 baar). Yahi p-value hai. Kya 17% itna rare hai ki kaho ki coin magic hai? Zyaadatar scientists kehte hain tumhe convince hone ke liye 5% se kam (really rare) chahiye. Toh is case mein, tum kehte, "Nah, 10 mein se 7 aasaani se sirf luck ho sakta hai. Mujhe aur flips dikhao!"

P-value tumhare data ke liye ek "weirdness score" jaisa hai. Jitna chhota hoga, utna zyada tum boring explanation par doubt karte ho aur believe karne lagte ho ki kuch special ho raha hai.

Connections

1.3.1-Random-variables-and-distributions – Test statistics known distributions follow karte hain (normal, chi-squared, t)
1.3.15-Central-limit-theorem – Kyun hum test statistics ke liye normal approximations use kar sakte hain
1.3.18-Confidence-intervals – Hypothesis testing ka dual: agar $μ_{0}$ 95% CI ke bahar hai, toh $α = 0.05$ par reject karo
1.3.21-Type-I-and-Type-II-errors – Hypothesis tests se control hone wale error rates
2.5.7-Statistical-significance-in-experiments – ML systems mein A/B tests par hypothesis testing apply karna
3.2.12-Multiple-testing-correction – Jab bahut saari hypotheses test karein, inflated false positives ke liye adjust karo (Bonferroni, FDR)

#flashcards/ai-ml

P-value kya hota hai? :: Yeh probability hai ki hum apna observed data utna hi extreme ya aur zyada extreme data observe karein, yeh maante hue ki null hypothesis $H_{0}$ sach hai. Yeh measure karta hai ki humara data $H_{0}$ ke under kitna "unusual" hai.

α = 0.05

par statistically significant" ka kya matlab hai?

P-value 0.05 se kam hai, toh hum null hypothesis reject karte hain.

H_{0}

ke under, itna extreme data 5% se kam time hoga.

Null hypothesis

H_{0}

kya hota hai?

Default "koi effect nahi" assumption jise hum test karte hain. Hum maante hain yeh sach hai jab tak data isko reject karne ka strong evidence na de. Example: "Coin fair hai," "Dono groups ke means equal hain."

Alternative hypothesis

H_{1}

kya hota hai?

Woh claim jiske liye hum evidence chahte hain, kehta hai ki effect ya difference IS hai. Example: "Coin biased hai," "Treatment group ka mean control se zyada hai."

H_{0}

reject karne mein fail hona" ka kya matlab hai?

P-value

\geq α

hai, toh hamare paas null ko reject karne ka itna strong evidence nahi hai. Iska matlab yeh NAHI ki

H_{0}

sach hai—sirf itna ki data uske saath consistent hai.

Two-tailed test mein p-value compute karte waqt hum 2 se kyun multiply karte hain?

Kyunki hum dono directions mein deviations ki care karte hain (bahut zyada ya bahut kam). Hum dono tails mein extreme hone ki probability count karte hain.

Hypothesis testing mein test statistic kya hota hai?

Ek number jo data se compute hota hai aur summarize karta hai ki observation

H_{0}

ki prediction se kitna door hai. Example: z-score

Z = \frac{X ˉ - μ _{0}}{σ / n}

Type I aur Type II errors mein kya difference hai?

Type I (false positive):

H_{0}

ko reject karna jab yeh sach ho, probability =

α

. Type II (false negative):

H_{0}

ko reject karne mein fail hona jab yeh galat ho, probability =

β

One-tailed vs two-tailed test kab use karte hain?

One-tailed: Jab sirf ek direction mein deviations ki care ho (jaise "naya model better hai"). Two-tailed: Jab kisi bhi difference ki care ho (jaise "naya model different hai").

Significance level

α

kya hota hai?

H_{0}

reject karne ka threshold. Agar

p < α

toh reject karo. Common choice: 0.05, matlab hum 5% false positive rate tolerate karte hain.

"p < 0.05 matlab

H_{0}

galat hai" yeh wrong interpretation kyun hai?

P-value hai

P (data ∣ H_{0})

P (H_{0} ∣ data)

nahi. Yeh

H_{0}

ke under data ki extremeness measure karta hai, na ki probability ki

H_{0}

galat hai.

Confidence intervals aur hypothesis tests mein kya relationship hai?

Agar koi value

μ_{0}

μ

ke 95% confidence interval ke bahar hai, toh tum

α = 0.05

par

H_{0} : μ = μ_{0}

reject karte. Yeh dono dual methods hain.

Hypothesis Testing Kya Hai?

p-value Derive Karna (Coin Example)

Step 1: Test Statistic Choose Karo

Step 2: p-value Compute Karo

Step 3: Decision

Formula Reference

Worked Examples

Common Mistakes

The Significance Level $α$

Active Recall Practice

Connections

Concept Map