Hypothesis testing and p-values: Question bank — Hinglish deep dive

1.3.20 · D5 · AI-ML › Probability & Statistics › Hypothesis testing and p-values

Intuition Is page ko kaise use karein

Neeche har line ek claim, error, ya edge case hai jo hypothesis testing aur p-values ke baare mein ek common misconception chhupa raha hai. Answer ko cover karo, decide karo, phir reveal karo. Answer side hamesha reasoning deti hai, sirf verdict nahi — wahi reasoning yaad karne layak hai.

In traps mein jo vocabulary use hoti hai, uska reminder:

$H_{0}$ — null hypothesis, "kuch interesting nahi ho raha" wala default jo hum testing ke dauran true maante hain.
$H_{1}$ — alternative hypothesis, woh effect jiske liye hum search kar rahe hain.
p-value — hamare data jaisa ya usse zyada extreme data milne ki probability, given ki $H_{0}$ true hai.
$α$ — significance level, pehle se chuna hua "itna strange ki reject kar do" threshold.
$n$ — sample size, kitne observations collect kiye (100 coin flips → $n = 100$ ).
$μ_{0}$ — null mean, woh exact value jo $H_{0}$ claim karta hai ki true average yahi hai (e.g. "coin ka heads-rate $0.5$ hai").
$σ$ — population standard deviation, individual observations ka true spread; $σ / n$ sample mean ka spread hai (standard error).
$Z$ — standard-normal test statistic, haara observation "kitne standard errors $μ_{0}$ se door uthaa" ke roop mein rewrite kiya: $Z = \frac{X ˉ - μ _{0}}{σ / n}$ . $H_{0}$ ke under yeh neeche wala bell curve follow karta hai, aur p-value hamare $Z$ ke baad shaded tail area hai.

True or false — justify karo

A p-value of 0.02 means there is a 2% chance the null hypothesis is true.

False. P-value

P (data ∣ H_{0})

hai, jo

H_{0}

ko true maanke compute kiya jaata hai; yeh kabhi

H_{0}

ki probability nahi ho sakta. Woh ulta quantity

P (H_{0} ∣ data)

ek prior aur Bayesian inference maangta hai.

A p-value of 0.60 proves the null hypothesis is correct.

False. Bada p-value ka sirf matlab hai ki data

H_{0}

ke under unsurprising hai; yeh

H_{0}

confirm nahi karta. Tum "fail to reject" karte ho, jo agnosticism hai, proof nahi — chhota ya noisy sample ek real effect chhhupa sakta hai.

If we reject

H_{0}

α = 0.05

, there is a 5% chance we made a mistake on this particular test.

False. 5% long-run rate hai false rejections ka jab

H_{0}

true ho many tests mein; ek specific rejection ke liye "error ki chance" 5% nahi hai aur prior ke bina defined bhi nahi hai.

Two studies with

p = 0.049

and

p = 0.051

carry essentially opposite scientific conclusions.

False. Evidence almost identical hai; sirf arbitrary

α = 0.05

line dono ke beech mein padti hai. Is cliff ko meaningful treat karna thresholds ka ek well-known abuse hai.

A statistically significant result (

p < 0.05

) is automatically important in practice.

False. Bahut bade sample size

n

ke saath, ek microscopic effect bhi threshold cross kar leta hai. Significance ka jawab hai "kya yeh noise se distinguishable hai?", "kya yeh care karne layak bada hai?" nahi — effect size check karo.

Lowering

α

from 0.05 to 0.01 makes the test better in every way.

False. Yeh false positives (Type I) kaata hai lekin false negatives (Type II) badhata hai, power ghataata hai. Yeh trade-off hai, free improvement nahi.

The p-value tells you how large the effect is.

False. P-value effect size aur sample size ko blend karta hai, toh yeh magnitude isolate nahi kar sakta. Bade

n

ke saath tiny effect aur chhote

n

ke saath huge effect same p-value de sakte hain.

A two-tailed p-value is always exactly twice the one-tailed p-value for the same data.

True symmetric distributions jaise normal ke liye, jahan har tail ka equal area hota hai, toh one-sided tail ko double karne se two-sided value milti hai — lekin sirf tab jab observed statistic tested side pe ho.

If your result is not significant, collecting more data until it becomes significant is a valid fix.

False. Baar baar peekh karna aur

p < 0.05

hone pe rokna true Type I rate ko 5% se kaafi upar inflate kar deta hai (optional stopping / p-hacking). Stopping rule pehle se fixed honi chahiye.

The confidence interval and the hypothesis test can disagree about the same parameter.

False, jab consistently build kiya jaaye: ek 95% confidence interval null mean

μ_{0}

ko exclude karta hai exactly tab jab two-tailed test

α = 0.05

pe reject karta hai. Yeh dono same computation ke do views hain.

Spot the error

"We got

p = 0.03

, so the probability our alternative hypothesis is true is 97%."

Error: p-values

P (H_{1} ∣ data)

ke baare mein kuch nahi kehte. 0.03

H_{0}

ke under compute kiya jaata hai; "97% true" wala claim transposed-conditional fallacy hai.

"We ran 20 independent A/B tests, and one came back with

p = 0.04

, so that feature clearly works."

Error: true nulls ke saath 20 tests ke saath tum

α = 0.05

pe lagbhag ek false positive expect karte ho. Celebrate karne se pehle multiple-testing correction chahiye.

H_{0}

is that the new model is better than the old one, and we test whether it's worse."

Error:

H_{0}

"no effect / status quo" claim hona chahiye (accuracy equal), aur

H_{1}

woh effect carry karta hai jiske liye tum evidence chahte ho. Yahan hypotheses swap ho gaye hain.

"Since the standardised statistic

Z = 1.9

and we only cared about improvement, we doubled the tail to get the p-value."

Error: one-tailed test ke liye tum single tail use karte ho (figure mein woh ek red region); doubling sirf two-tailed alternatives ke liye hai. Yahan double karna galat tarike se tumhare evidence ko aadha kar deta hai.

"We chose

α = 0.05

after seeing the data landed at

p = 0.048

Error:

α

outcome dekhne se pehle fix hona chahiye. Threshold ko apna p-value just clear karne ke liye choose karna p-hacking ka ek form hai.

"The die test gave

p = 0.33 > 0.05

, so we accept that the die is perfectly fair."

Error: tum fail to reject karte ho, jiska matlab hai "unfairness ka koi evidence nahi", "provably fair" nahi. Subtle bias sirf 60 rolls se undetectable ho sakta hai.

"We used a z-test on

n = 8

samples with unknown population standard deviation

σ

and a heavily skewed distribution."

Error: z-test Central Limit Theorem pe rely karta hai (bade

n

chahiye) aur known

σ

par. Chhote skewed samples aur estimated

σ

ke liye t-test aur caution chahiye.

Why questions

Why do we assume

H_{0}

is true before computing anything, instead of assuming

H_{1}

Kyunki

H_{0}

("no effect") probabilities compute karne ke liye ek exact distribution pin down karta hai, jabki

H_{1}

("some effect") ek vague family hai jiska koi single number nahi. "Yeh kitna weird hai?" sirf ek fixed reference ke against measure kar sakte ho.

Why is the p-value a tail probability ("at least as extreme") rather than the probability of exactly our observation?

Continuous data ke liye kisi exact value ki probability zero hai, aur discrete data ke liye bhi ek single point surprise ka poor measure hai. "At least as extreme" capture karta hai ki hum unlikely region mein kitna andar gaye, jo actually surprise ka matlab hai.

Why is 0.05 the usual

α

, and is it special?

Yeh special nahi hai — yeh ek 1920s ka convention hai (Fisher) jo convenience ke liye choose kiya gaya. Field ne ise standardise kar liya, lekin koi bhi threshold false positives ko false negatives ke against trade karta hai; "sahi"

α

har error ki cost pe depend karta hai.

Why does a larger sample size

n

shrink the p-value for a fixed true effect?

Zyada data standard error

σ / n

ko shrink karta hai (jahan

σ

population standard deviation hai), toh wahi real gap null mean

μ_{0}

se kaafi standard errors door ho jaata hai, test statistic ko tail mein push karta hai. Isliye huge

n

trivial effects ko "significant" bana deta hai.

Why must we report effect size alongside the p-value?

P-value akela ek meaningful effect aur ek trivial effect jo brute sample size se detect hui ho, mein fark nahi bata sakta. Effect size (ya ek confidence interval) "kitna?" ka jawab deta hai, woh question jo real experiments mein decisions drive karta hai.

Why does the chi-squared die test lose one degree of freedom (df = 5, not 6)?

Kyunki chhe observed counts ko fixed total (60) mein sum karna padta hai, toh ek baar paanch pata ho jaayein toh chhata forced hai. Ek linear constraint ek free dimension remove karta hai.

Why can rejecting

H_{0}

never prove

H_{1}

is exactly true?

Rejection sirf yeh kehta hai ki data

H_{0}

ke under implausibly extreme hai; tumhari specific

H_{1}

ke alawa kai alternatives bhi aisa data produce kar sakte hain. Ek claim ke against evidence kisi particular competing claim ka proof nahi hai.

Edge cases

What does the p-value equal if your standardised statistic

Z

sits exactly at the null mean (

Z = 0

Two-tailed test ke liye

p = 1

: data mean se kam extreme nahi ho sakta, toh essentially har outcome utna hi extreme hai. Yeh

H_{0}

ke saath maximal agreement hai.

If a coin gives exactly 50 heads in 100 flips, is that evidence the coin is fair?

Nahi — yeh fairness ke under most likely single outcome hai, p-value 1 ke karib deta hai, lekin "

H_{0}

ke saath consistent" matlab "

H_{0}

confirm" nahi hai. Ek cleverly biased ya two-headed-then-switched coin bhi yeh produce kar sakta hai.

What happens to power as the true effect size approaches zero?

Power

\to α

: jab effect gayab ho jaata hai, "use correctly detect karna" false-positive rate pe collapse ho jaata hai. Koi test reliably kuch aisa detect nahi kar sakta jo hai hi nahi.

Can a p-value ever be exactly 0?

Continuous theory mein nahi — normal tail hamesha positive hoti hai, toh kisi bhi finite statistic ke liye

p > 0

. Reported "

p = 0

" ka matlab hai "software ke rounding floor se chhota".

What is the p-value's distribution when

H_{0}

is exactly true?

Yeh

[0, 1]

pe Uniform hai: ek true null ke under har threshold

α

exactly

α

fraction of the time cross hota hai, jo isliye Type I rate

α

ke barabar hoti hai.

If two effects are each non-significant separately, can their combination be significant?

Haan — data pool karna ya evidence combine karna effective sample size raise kar sakta hai aur ek real-but-small joint effect ko threshold ke paas push kar sakta hai. Non-significance linearly add nahi hoti.

With one-tailed testing, what happens if the data land far in the unexpected direction?

One-tailed p-value 1 ke karib ho jaata hai (galat tail pe huge), toh tum reject karne mein fail ho jaate ho chahe result extreme ho — direction pehle se commit karne ki cost. Two-tailed test ise pakad leta.

Recall Ek-line self-check

Upar ke har answer ko cover karo; agar tum reasoning reconstruct kar sako (sirf true/false nahi), toh concept tumhara hai. Do sabse deadly traps jo zyada seekhne chahiye: p-value $\neq = P (H_{0} ∣ data)$ , aur "fail to reject" $\neq =$ "accept $H_{0}$ ".