4.9.24 · D5Probability Theory & Statistics
Question bank — Bayesian statistics — prior, likelihood, posterior (intro)
The picture above is the master image to carry through every question: prior curve (belief before), likelihood curve (how the data votes), and the sharpened posterior curve (belief after) — look at how the red posterior sits pulled between the other two.
True or false — justify
The prior can be any probability distribution I like, no data required.
True — the prior encodes belief before data, so it is chosen (from theory, past studies, or convenience), not derived from the current dataset. But "any" still must be a valid distribution that integrates to 1.
The likelihood is a probability distribution over .
False — is a function of but a probability of the data; it need not integrate to 1 over , so it is not a distribution over . See Maximum Likelihood Estimation which maximizes exactly this function.
If two hypotheses have the same likelihood for the observed data, the posterior treats them identically.
False — equal likelihoods only cancel the "persuasion" step; the posterior ratio then equals the prior ratio, so whichever hypothesis you believed more before still wins after.
The evidence affects the shape of the posterior over .
False — is a single number independent of , so it only rescales the whole curve to integrate to 1; it never changes which is more probable than another.
A flat (uniform) prior means "no assumptions / pure objectivity."
False — a prior flat in is not flat in a transform like or , so "flat" is a specific modelling choice, not the absence of one.
With enough data, the choice of prior stops mattering.
Usually true — as data accumulate the likelihood becomes sharply peaked and dominates a smooth prior, so posteriors from different reasonable priors converge. It fails if the prior assigns zero probability somewhere the truth lives.
and are just two notations for the same thing.
False — they condition on different things; with a 99% sensitive, 95% specific test but . Confusing them is the prosecutor's fallacy, and Bayes' theorem is precisely the bridge that flips the conditioning.
The posterior can be more concentrated (more certain) than both the prior and the likelihood.
True — multiplying two overlapping curves and renormalizing yields a product that is typically narrower than either factor, since agreement between prior and data sharpens belief.
The posterior mean always lies strictly between the prior mean and the data's estimate.
Generally true for conjugate cases like Beta–Binomial — it is a weighted average, so it sits between prior mean and sample fraction. It touches an endpoint only in the limiting cases (zero data, or infinitely strong prior/data).
Spot the error
"I got a positive test that is 99% sensitive and 95% specific, so I'm 99% likely to be sick."
The error swaps for and ignores the prior; with a rare disease, false positives from the huge healthy group swamp the true positives, giving far less than 99%.
"Posterior likelihood prior, so I never need at all."
The proportional form gives the correct shape, but to state a calibrated number like you must divide by . Omitting it is fine only for comparing -values, not for reporting an actual probability.
"My prior was for ; after tons of data suggesting , the posterior finally moved there."
Wrong — the posterior is proportional to prior times likelihood, and anything multiplied by a zero prior stays zero forever. A hypothesis given zero prior can never be resurrected by data.
"I computed the likelihood, saw it summed to over all , and concluded I made an arithmetic mistake."
No mistake — the likelihood is not required to integrate or sum to 1 over ; only the normalized posterior must. The "error" is expecting the likelihood to behave like a distribution over .
"The evidence , but I only summed over the hypotheses I like."
The sum must run over all hypotheses via the Law of Total Probability (and becomes an integral when is continuous); dropping some makes too small, so the posterior no longer normalizes and probabilities exceed 1.
"I updated on the same data twice to be sure, and my posterior got sharper."
Illegitimate — using one observation twice double-counts evidence, artificially inflating certainty. Each independent datum enters the likelihood once; reusing it violates the model that generated .
"To find the most likely , I took the peak of the likelihood, which must equal the posterior peak."
Only if the prior is flat over the relevant region — a non-uniform prior shifts the peak. The likelihood peak is the MLE; the posterior peak (MAP) generally differs.
Why questions
Why does a test that is 99% sensitive and 95% specific give only ~17% chance of disease at 1% prevalence?
Because there are ~100 times more healthy people than sick, and 5% of that huge group tests false-positive, so false positives numerically outnumber the true positives despite the high accuracy.
Why do we treat as a random variable at all, when a coin's bias is a fixed physical fact?
We aren't claiming physically fluctuates; the distribution encodes our uncertainty about the fixed value. This is the core Bayesian vs frequentist split — Bayesians model knowledge, not the object.
Why can we drop the binomial coefficient when finding the posterior shape?
It is a constant in , so it is absorbed into the normalizer and cancels; it affects the overall scale, never which is favoured.
Why is the Beta prior called "conjugate" to the Binomial likelihood?
Because multiplying a Beta prior by a Binomial likelihood yields another Beta, keeping the posterior in the same family. This makes updating a matter of adding counts rather than doing an integral — see Beta Distribution.
Why does a strong prior resist being changed by a single data point?
A sharply peaked prior gives near-zero weight away from its peak, so multiplying by a modest likelihood barely relocates the product's peak. Stubborn priors need overwhelming likelihood to shift.
Why does the Naive Bayes Classifier multiply many likelihoods together?
It assumes features are conditionally independent given the class, so the joint likelihood factorizes into a product of per-feature likelihoods — a modelling simplification that makes computation cheap even if only approximately true.
Why is called "evidence" and also "marginal likelihood"?
It is the likelihood marginalized over all (summed in the discrete case, integrated in the continuous case) — the average probability the model assigns to the data — so it measures how well the whole model predicted what was seen, useful for comparing models.
Edge cases
What is the posterior if the prior already puts all its mass on one (a spike)?
It stays a spike at that same — a degenerate prior of certainty is unmovable, because zero probability elsewhere multiplied by any likelihood remains zero.
What happens to the posterior when you observe zero data ( empty)?
The likelihood is constant (nothing to explain), so the posterior equals the prior — no data means no update, exactly as the recipe demands.
With a flat prior and 0 heads in 0 tosses, what is the posterior?
It remains , the uniform distribution — no observations means the prior passes through unchanged, so every bias stays equally plausible.
If the observed data has zero probability under every hypothesis, what does Bayes give?
, so the formula divides by zero and is undefined — this signals the model cannot generate the data at all, meaning the hypothesis space or model is misspecified.
You flip a coin once and see one head; what does say about ?
The posterior density at becomes zero, correctly ruling out a coin that never lands heads — a single head is impossible under , so the likelihood there is zero.
In the limit of infinite data, what does the Bayesian posterior mean converge to?
It converges to the true parameter value (under mild conditions), coinciding with the frequentist MLE — the prior's influence vanishes as the likelihood dominates.
Recall One-line self-test before you leave
Can you say, in one breath, which of prior/likelihood/posterior/evidence (a) need not integrate to 1, (b) is a single -independent number, (c) can be resurrected from zero, (d) survives with no data? ::: (a) likelihood; (b) evidence; (c) none — zero prior stays zero; (d) the prior (posterior = prior when no data).