Hypothesis testing — null - alternative, test statistic, p-value, errors (Type I & II): Question bank

Reminders of the words we lean on, all built in the parent:

$H_{0}$ = the boring "no effect" claim we assume true; $H_{1}$ = the claim we hunt evidence for.
$α$ = the false-alarm rate we choose in advance; $p$ = how surprising this data is if $H_{0}$ holds.
Type I = reject a true $H_{0}$ ; Type II = fail to reject a false $H_{0}$ ; power $= 1 - β$ .

True or false — justify

A small p-value proves the null hypothesis is false.

False. A small

p

only says the data is surprising if $H_{0}$ holds; it is

P (data ∣ H_{0})

, not

P (H_{0} ∣ data)

, so it can never "prove"

H_{0}

false — a rare fluke under a true

H_{0}

produces small

p

exactly

α

of the time.

p = 0.20

, there is a

20%

chance the null is true.

False.

p

is computed assuming

H_{0}

is already true; it says nothing about the probability of

H_{0}

. Turning

P (data ∣ H_{0})

into

P (H_{0})

is the prosecutor's fallacy — you'd need Bayesian Inference and a prior to talk about

P (H_{0} ∣ data)

A result with

p = 0.049

is meaningfully stronger evidence than one with

p = 0.051

False in any real sense. The

α = 0.05

line is a chosen convention, not a law of nature; the two data sets are almost identical, so treating one as "significant" and the other as "not" is an artefact of a hard threshold.

Failing to reject

H_{0}

means we have proven

H_{0}

is true.

False. "Fail to reject" means the evidence wasn't strong enough to convict — often just small

n

or low power. Absence of evidence is not evidence of absence; we say "insufficient evidence," never "proven."

Lowering

α

from

0.05

0.01

makes the test better in every way.

False. A stricter

α

pushes the critical cutoff farther out, cutting false alarms but raising

β

(Type II errors), so you miss more real effects. It's a trade, not a free upgrade — only larger

n

improves both.

The alternative hypothesis

H_{1}

can contain the equality (e.g.

μ = μ_{0}

False.

H_{0}

always carries the equality because we need one specific value to compute the sampling distribution and the p-value.

H_{1}

states

\neq =

>

, or

<

Power

1 - β

depends on how far the true mean actually is from

μ_{0}

True. The bigger the real effect, the more the alternative sampling distribution shifts away from

H_{0}

, so more of it lands past the critical cutoff and power rises. See Statistical Power & Sample Size.

Increasing sample size

n

shrinks both

α

and

β

False for

α

, true for

β

α

is a value we choose, so

n

doesn't touch it; but larger

n

shrinks the Standard Error, narrowing both distributions so

β

drops at the same

α

95%

confidence interval and a two-tailed test at

α = 0.05

always agree about

μ_{0}

True. Rejecting

H_{0} : μ = μ_{0}

α = 0.05

is equivalent to

μ_{0}

falling outside the

95%

confidence interval — they are two views of the same standardized distance.

Spot the error

"We got

p = 0.03 < 0.05

, so there's a

97%

chance our effect is real."

The error is reading

1 - p

as the probability the effect exists.

p

is conditional on

H_{0}

; the probability the effect is real needs a prior and Bayesian machinery. Rejecting

H_{0}

is a decision, not a probability statement about reality.

"Our test wasn't significant, so the drug has no effect."

Confuses "fail to reject" with "proven no effect." A null result may just reflect low power or small

n

; the true effect could be real but too small for this test to catch.

α = 0.05

means at most

5%

of all my published rejections are false discoveries."

α = P (reject ∣ H_{0} true)

— conditional on the null being true. The share of your rejections that are wrong (the false discovery rate) also depends on how often

H_{0}

is actually true, which

α

alone never tells you.

"The data leaned high, so I'll switch to a right-tailed test to get significance."

Choosing the tail after seeing the data doubles your real false-alarm rate — you effectively test both directions while claiming one. The direction must be fixed before looking at the data.

"We used

Z = (\overset{ˉ}{X} - μ_{0}) / (s / n)

with a sample of

n = 9

and treated it as standard normal."

When

σ

is unknown and estimated by

s

, the statistic follows a Student t-distribution with

n - 1

degrees of freedom, not

N (0, 1)

. For small

n

its heavier tails matter; using

z

understates the p-value.

"We ran

20

separate tests at

α = 0.05

and one came out significant, so we found something."

With

20

independent true nulls you expect about one false positive by chance (

20 \times 0.05 = 1

). A single hit out of many tests is exactly the false-alarm rate at work, not a discovery — you need a multiple-comparison correction.

"Our huge sample gave

p = 0.001

, so the effect is large and important."

A tiny p-value from a massive

n

can flag a trivially small effect — statistical significance is not practical significance. Large

n

shrinks the standard error so even a microscopic difference from

μ_{0}

becomes "detectable."

Why questions

Why must

H_{0}

always contain the equality rather than an inequality?

Because we need one exact value (

μ = μ_{0}

) to pin down a specific sampling distribution and compute the p-value; an inequality like

μ > μ_{0}

names infinitely many distributions and gives us nothing to standardize against.

Why do we double the tail probability for a two-tailed p-value?

"As or more extreme" in a two-sided test means far in either direction, so both symmetric tails count as evidence — hence

p = 2 P (Z \geq ∣ z_{o b s} ∣)

Why does the test statistic divide by the standard error instead of the standard deviation

σ

We're judging how far the sample mean

\overset{ˉ}{X}

sits from

μ_{0}

, and

\overset{ˉ}{X}

varies with spread

σ / n

(the Standard Error), not

σ

. Dividing by SE gives a unit-free count of "how many standard errors away."

Why does the Central Limit Theorem let us use the normal distribution even when the raw data isn't normal?

The CLT says the sampling distribution of $\overset{ˉ}{X}$ approaches normal as

n

grows, regardless of the population's shape — so the standardized statistic is approximately standard normal even for skewed data.

Why can't we ever "accept" the null hypothesis?

Because the test is designed only to detect deviations, not confirm their absence; a non-significant result is consistent with

H_{0}

and with many small real effects we lacked power to see. We can only fail to find enough evidence to reject.

Why is a Type I error controlled directly but a Type II error is not?

We choose

α

as the cutoff, so we set the Type I rate by design.

β

depends on the unknown true effect size,

n

, and

α

together, so it emerges rather than being dialed in directly.

Why does making a test more powerful usually require more data rather than just a lower

α

Lowering

α

actually reduces power. Power grows by separating the two sampling distributions and narrowing them — a larger

n

shrinks the standard error and does both, whereas

α

only moves the cutoff.

Edge cases

If the observed sample mean lands exactly on

μ_{0}

, what is the two-tailed p-value?

z_{o b s} = 0

, so

p = 2 P (Z \geq 0) = 2 (0.5) = 1

. The data is the least surprising possible under

H_{0}

— you never reject.

What happens to the t-statistic's distribution as

n \to \infty

The Student t-distribution with

n - 1

degrees of freedom converges to the standard normal, because

s

becomes a near-perfect estimate of

σ

and the extra tail-uncertainty vanishes.

If you set

α = 0

to eliminate all false alarms, what happens to your ability to detect real effects?

The critical cutoff moves to infinity, so you can never reject

H_{0}

— Type I error is zero but

β = 1

and power is zero. You've built a test that always acquits, useless for finding anything.

If you set

α = 1

, what does the test do?

You reject

H_{0}

for any data whatsoever; power is

1

but every true null is falsely rejected. The two extremes (

α = 0

and

α = 1

) show why

α

is a deliberate compromise.

For a one-sided right-tailed test, what if the sample mean falls below

μ_{0}

The evidence points opposite to

H_{1} : μ > μ_{0}

, so

z_{o b s} < 0

and

p = P (Z \geq z_{o b s}) > 0.5

. You cannot reject — a result contradicting your alternative can never support it.

What if two different analysts pick

α = 0.05

and

α = 0.01

on the same data with

p = 0.03

The first rejects

H_{0}

, the second fails to reject — both correct given their chosen risk tolerance. The p-value is fixed by the data; the verdict depends on the threshold each chose beforehand.

As the true effect size shrinks toward zero, what happens to the power of a fixed test?

Power falls toward

α

. When there's essentially no effect, the alternative distribution sits almost on top of

H_{0}

, so the only rejections are the false alarms occurring at rate

α

Recall One-line self-check

Cover every answer above; if any verdict came out as a bare "true/false" with no reason, you haven't earned it yet — the reasoning is the concept.