Hypothesis testing — null - alternative, test statistic, p-value, errors (Type I & II)

WHY does hypothesis testing exist?

We have a sample, but we want to make a decision about the whole population. The sample is noisy: even if nothing special is happening, random fluctuation produces patterns. So we need a disciplined rule that controls how often we get fooled by randomness. Hypothesis testing is exactly that rule.

WHAT is a test statistic? A single number computed from the data that measures how far the sample sits from what $H_0$ predicts, measured in units of standard error so we know its random distribution.

HOW to derive the z-test statistic from first principles

We want to test $H_0:\mu=\mu_0$ using sample mean $\bar X$ from $n$ observations with known population SD $\sigma$.

Step 1 — distribution of $\bar X$. Each $X_i$ has mean $\mu$ and variance $\sigma^2$. The sample mean is $\bar X=\frac1n\sum X_i$.

$\mathbb E[\bar X]=\mu,\qquad \operatorname{Var}(\bar X)=\frac{1}{n^2}\sum\operatorname{Var}(X_i)=\frac{\sigma^2}{n}.$ Why this step? Variance of a sum of independent variables adds; the $1/n^2$ comes from the constant factor squared.

Step 2 — standard error. So the spread of $\bar X$ is $\operatorname{SE}=\sigma/\sqrt n$. Why? SD is the square root of variance.

Step 3 — standardize under $H_0$. Assume $H_0$ true, so $\mu=\mu_0$. By the Central Limit Theorem, $\bar X$ is approximately normal, hence

$\boxed{\;Z=\dfrac{\bar X-\mu_0}{\sigma/\sqrt n}\;}\sim \mathcal N(0,1).$ Why this step? Subtract the assumed mean, divide by SE → a unit-free score telling us how many standard errors the data is from $H_0$.

p-value: the probability of being fooled this badly

The two ways to be wrong

WHY there's a tradeoff: lowering $\alpha$ (being stricter about rejecting) pushes the critical cutoff farther out, which makes it harder to detect true effects → $\beta$ rises. The only way to shrink both is to increase $n$ (narrows both distributions).

Worked Example 1 — z-test, two-tailed

A machine should fill bottles to $\mu_0=500$ ml, $\sigma=8$ ml. We sample $n=64$ bottles, get $\bar X=497$ ml. Test at $\alpha=0.05$ whether the machine is off.

• $H_0:\mu=500$, $H_1:\mu\ne 500$. Why two-tailed? "Off" means either too high or too low.
• $\operatorname{SE}=8/\sqrt{64}=1$. Why? $\sigma/\sqrt n$.
• $z=\dfrac{497-500}{1}=-3$. Why? Standardize the observed mean.
• $p=2P(Z\ge 3)\approx 2(0.00135)=0.0027$. Why doubled? Both tails are "as extreme."
• $0.0027<0.05$ → reject $H_0$. The machine is mis-calibrated.

Worked Example 2 — right-tailed t-test

A new teaching method claims to raise scores above the old mean $\mu_0=70$. Sample: $n=16$, $\bar X=74$, $s=6$. Test at $\alpha=0.05$.

• $H_0:\mu=70$, $H_1:\mu>70$. Why right-tailed? Claim is only "higher."
• $T=\dfrac{74-70}{6/\sqrt{16}}=\dfrac{4}{1.5}=2.667$ with $df=15$. Why $t$ not $z$? $\sigma$ unknown.
• Critical $t_{0.05,15}\approx1.753$. Since $2.667>1.753$, $p<0.05$ → reject $H_0$: evidence the method helps.

Worked Example 3 — interpreting a non-rejection

Same as Ex.2 but $\bar X=71$: $T=\frac{1}{1.5}=0.667<1.753$ → fail to reject. Why this wording? We did not prove the method useless; we just lack evidence. Absence of evidence ≠ evidence of absence.

Active Recall

Connections

• Central Limit Theorem — justifies the normal distribution of $\bar X$.
• Standard Error — the denominator of every test statistic.
• Confidence Intervals — dual of two-tailed tests (reject ⇔ $\mu_0$ outside CI).
• Normal Distribution and Student t-distribution — reference distributions.
• Bayesian Inference — gives $P(H_0\mid \text{data})$ directly (what p-values don't).
• Statistical Power & Sample Size — controlling $\beta$.

Concept Map

Hinglish (regional understanding)