Exercises — Hypothesis testing and p-values

The null is always the boring, no-effect, equality statement. (a) $H_0:p=0.5$ (fair). $H_1:p\ne 0.5$ (biased) — two-sided because "biased" could go either way. (b) $H_0:p=0.82$ (no improvement). $H_1:p>0.82$ (better) — one-sided because we only care about better.

Rule: reject when $p<\alpha$.

• $p=0.02<0.05$ → reject $H_0$.
• $p=0.20\ge 0.05$ → fail to reject $H_0$ (not "accept").

(B). The p-value is $P(\text{data}\mid H_0)$, never $P(H_0\mid \text{data})$. Statement (A) reverses the conditioning.

Under $H_0$, $X\sim \text{Binomial}(400,0.5)$, so $\mu =np=400\cdot 0.5=200,\sigma =\sqrt{np(1-p)}=\sqrt{400\cdot 0.5\cdot 0.5}=\sqrt{100}=10.$ The z-statistic counts standard deviations from the mean: $Z=\frac{X-\mu }{\sigma }=\frac{220-200}{10}=\mathrm{2.0.}$

Two-tailed because $H_1:p\ne 0.5$ counts extremeness on both sides: $p=2\cdot P(Z\ge 2)=2\cdot 0.02275=\mathrm{0.0455.}$ Since $0.0455<0.05$, reject $H_0$ — but only just. Look at the figure below: our $Z=2$ lands right at the edge of the shaded region.

Standardise the observed proportion using the null's standard error: ${\sigma }_{\hat{p}}=\sqrt{\frac{p_0(1-p_0)}{n}}=\sqrt{\frac{0.80\cdot 0.20}{150}}=\sqrt{\frac{0.16}{150}}=\sqrt{0.0010667}\approx \mathrm{0.03266.}$ $Z=\frac{\hat{p}-p_0}{{\sigma }_{\hat{p}}}=\frac{0.86-0.80}{0.03266}\approx \mathrm{1.837.}$ One-tailed p-value: $p=P(Z\ge 1.837)\approx 0.0331$. Since $0.0331<0.05$, reject $H_0$ — the model is significantly better.

One-tailed p-value: $p=P(Z\ge 2)=0.02275$. That is exactly half the two-tailed value. Decision: reject (still $<0.05$, now more comfortably). What it reveals: the same data can be "more significant" simply by choosing one tail. This is why you must pick one-vs-two-tailed before seeing the data — otherwise you can nudge yourself past the threshold. Halving the p-value by switching tails after the fact is a form of p-hacking.

${\sigma }_{\hat{p}}=\sqrt{0.5\cdot 0.5/n}=0.5/\sqrt{n}$. Lab A: $\sigma =0.5/\sqrt{100}=0.05$, so $Z=(0.55-0.5)/0.05=1.0$. Two-tailed $p=2P(Z\ge 1)\approx 0.317$ → fail to reject. Lab B: $\sigma =0.5/\sqrt{10000}=0.005$, so $Z=(0.55-0.5)/0.005=10.0$. $p$ astronomically small → reject. Comment: the effect ($0.55$ vs $0.5$) is identical, but larger $n$ shrinks the standard error, so the same effect becomes overwhelmingly "significant." Statistical significance is not the same as a large or important effect — it grows with sample size (see the drug example in the parent's Mistake 3).

Each term $(O_i-E_i{)}^2/E_i=(10-10{)}^2/10=0$, so ${\chi }^2=0$. A perfectly-fitting sample is as un-extreme as possible, so the tail probability $P({\chi }_5^2\ge 0)=1$, i.e. $p=1$. Interpretation: $p=1$ does not prove the die is fair — it just means the data is maximally consistent with fairness. "Consistent with $H_0$" is never "$H_0$ proven."

(a) $z_{\text{crit}}$ is the value putting $\alpha /2=0.025$ in each tail: $P(Z\ge 1.96)=0.025$, so together the two tails hold exactly $\alpha =0.05$. (b) The two-tailed test rejects when $\left|\frac{\bar{X}-{\mu }_0}{\sigma /\sqrt{n}}\right|\ge 1.96⟺|\bar{X}-{\mu }_0|\ge 1.96\frac{\sigma }{\sqrt{n}}.$ The $95\%$ confidence interval is $\bar{X}\pm 1.96\sigma /\sqrt{n}$. It excludes ${\mu }_0$ exactly when $|\bar{X}-{\mu }_0|>1.96\sigma /\sqrt{n}$ — the same inequality. So a two-sided test at level $\alpha$ and a $(1-\alpha )$ confidence interval are two views of one machine (see 1.3.18-Confidence-intervals).

$\alpha$ is the Type I error rate (false positive). Since false positives are the expensive mistake, lower $\alpha$ (e.g. $0.01$). Trade-off: lowering $\alpha$ raises $\beta$ (Type II, false negatives) and lowers power $1-\beta$ — but here false negatives are cheap, so that's acceptable. See 1.3.21-Type-I-and-Type-II-errors.

Reject when $\hat{p}\ge p_0+1.645\sqrt{p_0(1-p_0)/n}=0.5+1.645(0.05)=0.582$ (using null SE $=0.05$). Power $=P_{H_1}(\hat{p}\ge 0.582)$. Under $H_1$, SE $=\sqrt{0.6\cdot 0.4/100}=\sqrt{0.0024}=0.04899$. $Z^{\prime }=\frac{0.582-0.6}{0.04899}=\frac{-0.018}{0.04899}\approx -\mathrm{0.3675.}$ Power $=P(Z^{\prime }\ge -0.3675)=P(Z^{\prime }\le 0.3675)\approx 0.6434$. So this test has about a 64% chance of catching a genuinely biased ($p=0.6$) coin. Not great — you'd want a larger $n$ to raise power.

Each test has a $0.05$ chance of a false positive, so a $0.95$ chance of no false positive. Independent, so: $P(\text{at least one false positive})=1-0.95^{20}=1-0.3585=\mathrm{0.6415.}$ About a 64% chance you "discover" something that isn't real! This is the multiple-comparisons problem. Fix: a multiple-testing correction such as Bonferroni — test each at $\alpha /m=0.05/20=0.0025$. See 3.2.12-Multiple-testing-correction.

Per-test survival $=1-0.0025=0.9975$. $P(\text{at least one})=1-0.9975^{20}=1-0.9512=0.0488<\mathrm{0.05.}$ Yes — Bonferroni pulls the family-wide false-positive rate back under $0.05$, at the cost of lower power on each individual test.

"The tiny p-value only means the $0.5\%$ effect is very unlikely to be pure chance — with a large enough trial even trivial effects reach significance. The effect size itself ($0.5\%$, CI $[0.2\%,0.8\%]$) is clinically negligible, so 'statistically significant' here does not mean 'clinically important' — statistical and practical significance are different things (see 2.5.7-Statistical-significance-in-experiments)."

When is a p-value two-tailed? ::: When $H_1$ is "$\ne$" — extreme in either direction, so you double the one-tail area. Reject or not if $p=0.0455$, $\alpha =0.05$? ::: Reject (just barely, since $0.0455<0.05$). Why does the same effect get "more significant" with bigger $n$? ::: Standard error shrinks like $1/\sqrt{n}$, so $Z$ grows. What does power measure? ::: $1-\beta$, the chance of correctly rejecting a false $H_0$. Bonferroni per-test level for 20 tests at family $\alpha =0.05$? ::: $0.05/20=0.0025$.