Hypothesis testing and p-values

What is Hypothesis Testing?

Hypothesis testing is a statistical framework to decide between two competing claims about a population, using sample data.

The Process (from first principles):

Assume $H_0$ is true (innocent until proven guilty).
Compute a test statistic from your data—a number summarizing how far your observation is from what $H_0$ predicts.
Calculate the p-value: $P(\text{test statistic as extreme as observed} \mid H_0 \text{ true})$ .
Decision rule: If $p < \alpha$ (significance level, often 0.05), reject $H_0$ . Otherwise, fail to reject it.

Deriving the p-value (Coin Example)

Setup: You flip a coin $n = 100$ times, observe $X = 65$ heads. Test $H_0: p = 0.5$ vs $H_1: p \neq 0.5$ .

Step 1: Choose a Test Statistic

Under $H_0$ , $X \sim \text{Binomial}(n=100, p=0.5)$ . For large $n$ , use the normal approximation:

Z = \frac{X - \mu}{\sigma} = \frac{X - np}{\sqrt{np(1-p)}}

Why? The CLT says sample proportions are approximately normal. $Z$ measures "how many standard deviations away from the mean."

Derivation:

Mean: $\mu = np = 100 \cdot 0.5 = 50$
Std dev: $\sigma = \sqrt{np(1-p)} = \sqrt{100 \cdot 0.5 \cdot 0.5} = 5$
Observed: $Z = \frac{65 - 50}{5} = 3$

Step 2: Compute the p-value

Since $H_1: p \neq 0.5$ is two-tailed (we care about bias in either direction), the p-value is:

p\text{-value} = P(|Z| \geq 3 \mid H_0) = 2 \cdot P(Z \geq 3)

Why the2? We count both tails: $Z \geq 3$ (too many heads) and $Z \leq -3$ (too many tails).

From the standard normal table: $P(Z \geq 3) \approx 0.0013$

p\text{-value} = 2 \cdot 0.0013 = 0.0026

Step 3: Decision

$p = 0.0026 < 0.05 = \alpha$ → Reject $H_0$ . Conclusion: Strong evidence the coin is biased.

What the p-value means: If the coin were fair, there's only a 0.26% chance we'd see a result this extreme (or more). That's rare enough to doubt the "fair coin" assumption.

Worked Examples

Worked example Example 1: A/B Testing a Model

Scenario: Your old model has 82% accuracy. A new model gets 87% on 200 test samples. Is the improvement real or just luck?

Setup:

$H_0$ : New model accuracy = 82% (no improvement)
$H_1$ : New model accuracy > 82% (one-tailed, we only care if it's better)
Observed: $\hat{p} = 0.87$ , $n = 200$ Step 1: Test Statistic Under $H_0$ , accuracy follows $\text{Binomial}(200, 0.82)$ . Normal approximation: $Z = \frac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}} = \frac{0.87 - 0.82}{\sqrt{0.82 \cdot 0.18 / 200}}$

Why this step? We're standardizing the difference to measure how unusual it is.

$Z = \frac{0.05}{\sqrt{0.1476/200}} = \frac{0.05}{0.0272} \approx 1.84$

Step 2: p-value (one-tailed) $p = P(Z \geq 1.84) \approx 0.033$

Why one-tailed? We don't care if the new model is worse; we only reject if it's better.

Step 3: Decision $p = 0.033 < 0.05$ → Reject $H_0$ . The improvement is statistically significant.

Worked example Example 2: Testing if a Die is Fair

You roll a die 60 times, get these counts:

Face	1	2	3	4	5	6
Count	5	8	12	15	10	10

Question: Is the die fair?

Setup: $H_0$ : All faces equally likely (each $p = 1/6$ ). Use a chi-squared test.

$\chi^2 = \sum_{i=1}^{6} \frac{(O_i - E_i)^2}{E_i}$

Why this formula? It measures how far observed counts ( $O_i$ ) deviate from expected counts ( $E_i = 60/6 = 10$ ) under $H_0$ .

Calculation: $\chi^2 = \frac{(5-10)^2}{10} + \frac{(8-10)^2}{10} + \frac{(12-10)^2}{10} + \frac{(15-10)^2}{10} + \frac{(10-10)^2}{10} + \frac{(10-10)^2}{10}$ $= \frac{25 + 4 + 4 + 25 + 0 + 0}{10} = 5.8$

Why this step? Each term is squared to penalize deviations in both directions, divided by expected to standardize.

Degrees of freedom: $df = 6 - 1 = 5$ (Why? One constraint: total must sum to 60). From chi-squared table: $P(\chi^2_{5} \geq 5.8) \approx 0.33$

Decision: $p = 0.33 > 0.05$ → Fail to reject $H_0$ . Not enough evidence to say the die is unfair.

Common Mistakes

The Significance Level $\alpha$

What is $\alpha$ ? The significance level is the threshold for "weird enough to reject." Commonly0.05, but it's arbitrary.

Where does 0.05 come from? Historical convention (R.A. Fisher, 1920s). It balances two errors:

Type I error ( $\alpha$ ): Rejecting $H_0$ when it's true (false positive)
Type II error ( $\beta$ ): Failing to reject $H_0$ when it's false (false negative)

\text{Power} = 1 - \beta = P(\text{reject } H_0 \mid H_1 \text{ true})

Trade-off: Lower $\alpha$ → fewer false positives, but more false negatives (less power).

Active Recall Practice

Recall Feynman Explanation (Explain to a 12-year-old)

Okay, imagine your friend says, "I have a magic coin that always lands on heads!" You're skeptical, so you say, "Prove it." They flip it 10 times, and it lands on heads 7 times.

Now, here's the question: Is the coin really magic, or did they just get lucky?

Hypothesis testing is like being a detective. You start by assuming the coin is normal (the "null hypothesis"). Then you ask, "If the coin were normal, how often would I see 7 or more heads out of 10?" You calculate the chance—turns out it's about 17% (roughly 1 in 6 times). That's the p-value. Is 17% rare enough to say the coin is magic? Most scientists say you need less than 5% (really rare) to be convinced. So in this case, you'd say, "Nah, 7 out of 10 could easily just be luck. Show me more flips!"

The p-value is like a "weirdness score" for your data. The smaller it is, the more you doubt the boring explanation and start believing something special is happening.

Connections

1.3.1-Random-variables-and-distributions – Test statistics follow known distributions (normal, chi-squared, t)
1.3.15-Central-limit-theorem – Why we can use normal approximations for test statistics
1.3.18-Confidence-intervals – Dual to hypothesis testing: if $\mu_0$ is outside the95% CI, reject at $\alpha=0.05$
1.3.21-Type-I-and-Type-II-errors – The error rates controlled by hypothesis tests
2.5.7-Statistical-significance-in-experiments – Applying hypothesis testing to A/B tests in ML systems
3.2.12-Multiple-testing-correction – When testing many hypotheses, adjust for inflated false positives (Bonferroni, FDR)

#flashcards/ai-ml

What is a p-value? :: The probability of observing data at least as extreme as what we got, assuming the null hypothesis $H_0$ is true. It measures how "unusual" our data is under $H_0$ .

What does "statistically significant at

\alpha = 0.05

" mean?

The p-value is less than 0.05, so we reject the null hypothesis. Under

H_0

, data this extreme would occur less than 5% of the time.

What is the null hypothesis

H_0

The default "no effect" assumption we test against. We assume it's true unless data provides strong evidence to reject it. Example: "The coin is fair," "The two groups have equal means."

What is the alternative hypothesis

H_1

The claim we want evidence for, stating there IS an effect or difference. Example: "The coin is biased," "Treatment group has higher mean than control."

What does "fail to reject

H_0

" mean?

The p-value is ≥

\alpha

, so we don't have strong enough evidence to reject the null. This does NOT prove

H_0

is true—just that data is consistent with it.

In a two-tailed test, why do we multiply by 2 when computing the p-value?

Because we care about deviations in both directions (too high or too low). We count the probability of being extreme in either tail.

What is the test statistic in hypothesis testing?

A number computed from the data that summarizes how far the observation is from what

H_0

predicts. Example: z-score

Z = \frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}}

What is the difference between Type I and Type II errors?

Type I (false positive): Rejecting

H_0

when it's true, probability =

\alpha

. Type II (false negative): Failing to reject

H_0

when it's false, probability =

\beta

When do you use a one-tailed vs two-tailed test?

One-tailed: You only care about deviations in one direction (e.g., "new model is better"). Two-tailed: You care about any difference (e.g., "new model is different").

What is the significance level

\alpha

The threshold for rejecting

H_0

. If

p < \alpha

, we reject. Common choice: 0.05, meaning we tolerate a 5% false positive rate.

Why is "p < 0.05 means

H_0

is false" a wrong interpretation?

The p-value is

P(\text{data} \mid H_0)

, not

P(H_0 \mid \text{data})

. It measures data extremeness under

H_0

, not the probability that

H_0

is false.

What is the relationship between confidence intervals and hypothesis tests?

If a value

\mu_0

is outside the 95% confidence interval for

\mu

, you'd reject

H_0: \mu = \mu_0

\alpha = 0.05

. They're dual methods.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Chalo ise ek simple example se samajhte hain. Socho tumhare paas ek coin hai aur tum 100 baar flip karke 65 heads paate ho. Ab sawaal ye hai — kya coin fair hai ya rigged? Hypothesis testing basically ek framework hai jo poochta hai: "Agar coin sach mein fair hoti, toh itna weird result aane ke kitne chances hote?" Yahi weirdness ko measure karta hai p-value. Hum shuru mein maan lete hain ki coin fair hai (isko bolte hain null hypothesis, $H_0$ ), phir dekhte hain ki hamara actual data isse kitna dur hai. Iske liye ek test statistic nikalte hain (jaise $Z$ score) jo batata hai ki hamara result mean se kitne standard deviations door hai. Coin wale case mein $Z = 3$ nikla, aur p-value aaya 0.0026 — matlab agar coin fair hoti toh sirf 0.26% chance tha itna extreme result aane ka. Ye itna rare hai ki hum $H_0$ ko reject kar dete hain aur bolte hain coin biased hai.

Yahaan core intuition ye samajhna hai ki p-value tumhe seedha ye nahi batata ki coin biased hai ya nahi — ye batata hai ki agar default assumption sach hoti toh tumhara data kitna surprising hota. Chota p-value (usually 0.05 se kam) matlab data itna weird hai ki default belief pe shaq karna banta hai. Ek important cheez — two-tailed vs one-tailed. Agar tumhe dono direction ka bias check karna hai (zyada heads ya zyada tails), toh two-tailed test lagega aur p-value ko 2 se multiply karte hain. Lekin agar tumhe sirf ek direction check karni hai (jaise "kya new model behtar hai"), toh one-tailed test kaafi hai.

Ye concept AI-ML mein bahut kaam aata hai, isliye ise dil se samajh lo. Jab bhi tum A/B testing karte ho — jaise purana model 82% accuracy de raha tha aur naya 87% de raha hai — tumhe decide karna padta hai ki ye improvement genuine hai ya bas luck ya noise. Hypothesis testing tumhe ye confidently keh paane ki power deta hai ki "haan, ye result significant hai, sirf randomness nahi." Bina iske tum kisi bhi model improvement ya feature ke bare mein sahi decision nahi le paoge. Basically ye p-value ek bridge hai jo tumhare observed data aur uncertainty ke beech decisions lene mein madad karta hai — aur real-world ML mein har jagah yahi use hota hai.

What is Hypothesis Testing?

Deriving the p-value (Coin Example)

Step 1: Choose a Test Statistic

Step 2: Compute the p-value

Step 3: Decision

Formula Reference

Worked Examples

Common Mistakes

The Significance Level $\alpha$

Active Recall Practice

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability & Statistics

Connections