Question bank — Hypothesis testing and p-values

False. The p-value is $P(\text{data}\mid H_0)$, computed assuming $H_0$ is true; it can never be the probability of $H_0$. That flipped quantity $P(H_0\mid \text{data})$ needs a prior and Bayesian inference.

False. A large p-value only means the data are unsurprising under $H_0$; it never confirms $H_0$. You "fail to reject", which is agnosticism, not proof — a small or noisy sample can hide a real effect.

False. The 5% is the long-run rate of false rejections when $H_0$ is true across many tests; for one specific rejection the "chance of error" isn't 5% and isn't even defined without a prior.

False. The evidence is nearly identical; only the arbitrary $\alpha =0.05$ line falls between them. Treating this cliff as meaningful is a well-known abuse of thresholds.

False. With a huge sample size $n$, even a microscopic effect crosses the threshold. Significance answers "is it distinguishable from noise?", not "is it big enough to care about?" — check the effect size.

False. It cuts false positives (Type I) but raises false negatives (Type II), lowering power. It's a trade-off, not a free improvement.

False. The p-value blends effect size and sample size, so it can't isolate magnitude. A tiny effect with big $n$ and a huge effect with small $n$ can give the same p-value.

True for symmetric distributions like the normal, where each tail has equal area, so doubling the one-sided tail gives the two-sided value — but only when the observed statistic lies on the tested side.

False. Repeatedly peeking and stopping when $p<0.05$ inflates the true Type I rate far above 5% (optional stopping / p-hacking). The stopping rule must be fixed in advance.

False, when built consistently: a 95% confidence interval excludes the null mean ${\mu }_0$ exactly when the two-tailed test rejects at $\alpha =0.05$. They are two views of the same computation.

Error: p-values say nothing about $P(H_1\mid \text{data})$. The 0.03 is computed under $H_0$; the "97% true" claim is the transposed-conditional fallacy.

Error: with 20 tests under true nulls you expect about one false positive at $\alpha =0.05$. This needs multiple-testing correction before celebrating.

Error: $H_0$ must be the "no effect / status quo" claim (accuracy equal), and $H_1$ carries the effect you want evidence for. Here the hypotheses are swapped.

Error: for a one-tailed test you use a single tail (the one red region in the figure); doubling is only for two-tailed alternatives. Doubling here wrongly halves your evidence.

Error: $\alpha$ must be fixed before looking at the outcome. Choosing the threshold to just clear your p-value is a form of p-hacking.

Error: you fail to reject, which means "no evidence of unfairness", not "provably fair". A subtle bias could simply be undetectable with only 60 rolls.

Error: the z-test relies on the Central Limit Theorem (needs larger $n$) and a known $\sigma$. Small skewed samples with an estimated $\sigma$ call for a t-test and caution.

Because $H_0$ ("no effect") pins down an exact distribution to compute probabilities from, while $H_1$ ("some effect") is a vague family with no single number. You can only measure "how weird is this?" against a fixed reference.

For continuous data the probability of any exact value is zero, and even for discrete data a single point is a poor measure of surprise. "At least as extreme" captures how far into the unlikely region we landed, which is what surprise really means.

It isn't special — it's a 1920s convention (Fisher) chosen for convenience. The field standardised on it, but any threshold trades false positives against false negatives; the "right" $\alpha$ depends on the cost of each error.

More data shrinks the standard error $\sigma /\sqrt{n}$ (with $\sigma$ the population standard deviation), so the same real gap becomes many standard errors away from the null mean ${\mu }_0$, pushing the test statistic further into the tail. This is why huge $n$ makes trivial effects "significant".

The p-value alone can't tell a meaningful effect from a trivial one detected by brute sample size. Effect size (or a confidence interval) answers "how much?", the question that actually drives decisions in real experiments.

Because the six observed counts must sum to the fixed total (60), so once five are known the sixth is forced. One linear constraint removes one free dimension.

Rejection only says the data are implausibly extreme under $H_0$; many alternatives besides your specific $H_1$ could also produce such data. Evidence against one claim is not proof of a particular competing claim.

For a two-tailed test $p=1$: data can't be less extreme than the mean itself, so essentially every outcome is at least as extreme. This is maximal agreement with $H_0$.

No — it's the most likely single outcome under fairness, giving a p-value near 1, but "consistent with $H_0$" is not "confirms $H_0$". A cleverly biased or two-headed-then-switched coin could also produce it.

Power $\to \alpha$: when the effect vanishes, "correctly detecting it" collapses to the false-positive rate. No test can reliably detect an effect that isn't there.

Not in continuous theory — the normal tail is always positive, so $p>0$ for any finite statistic. Reported "$p=0$" really means "smaller than the software's rounding floor".

It is Uniform on $[0,1]$: under a true null every threshold $\alpha$ is crossed exactly a fraction $\alpha$ of the time, which is why the Type I rate equals $\alpha$.

Yes — pooling data or combining evidence can raise the effective sample size and push a real-but-small joint effect past the threshold. Non-significance doesn't add up linearly.

The one-tailed p-value approaches 1 (huge on the wrong tail), so you fail to reject even for an extreme result — the cost of committing to a direction in advance. A two-tailed test would have caught it.