1.3.3 · D5Probability & Statistics

Question bank — Bayes' theorem and applications

1,285 words6 min readBack to topic

Before you start, hold three phrases in mind, because almost every trap is a slip between them:

  • Prior — belief before evidence.
  • Likelihood — how well the cause explains the evidence.
  • Posterior — belief after evidence.

True or false — justify

Is always equal to ?
False. These are different questions ("A given B" vs "B given A") and are only equal in the special case ; the whole point of Bayes is that they usually differ.
A test with 99% accuracy means a positive result gives 99% chance of having the disease.
False. Accuracy is a likelihood ; the posterior also depends on the prior (base rate). A rare disease can still give a low posterior after a positive test.
The posterior can never be smaller than the prior.
False. Evidence against (evidence more likely under ) pushes the posterior below the prior; Bayes updates in both directions.
If the likelihood equals the likelihood , the evidence changes nothing.
True. When is equally likely whether or not holds, it fails to distinguish the hypotheses, so the posterior equals the prior — the evidence is uninformative.
A very accurate test is useless for a very rare condition.
Mostly true in spirit. When the prior is tiny, even a small false-positive rate produces far more false positives than true positives, so a single positive test barely raises the posterior — you need a second independent test.
Bayes' theorem is a new law of probability separate from conditional probability.
False. It is pure algebra: rearranging the two definitions of from conditional probability. It introduces no new axiom.
The denominator is the same for every hypothesis when comparing .
True. does not depend on which you ask about, so it is a shared normalizing constant — this is why it cancels in ratios of posteriors.
In Naive Bayes, the features really are independent given the class.
False. "Naive" flags the assumption as usually wrong; it is a tractability choice. The classifier often still ranks classes correctly despite the false assumption.

Spot the error

", so posterior equals likelihood over evidence."
The prior is missing from the numerator. The correct form is — dropping silently assumes .
"."
This only counts the world where is true. The law of total probability needs both branches: .
"Numerator and , so the posterior is ."
A probability above 1 is impossible, signalling a wrong . Since always, a numerator of forces ; the arithmetic exposes a mis-computed denominator.
"The likelihoods and must add to 1."
They need not. and sum to 1 (same condition), but and are probabilities under different conditions and are unrelated.
"Posterior of spam is , so of emails with 'FREE' are misclassified."
The is the probability a particular email is spam given it says "FREE", not an error rate over a population. Confusing a single-event posterior with a long-run frequency is a category error.
"Because Naive Bayes' probabilities are miscalibrated, its predictions are worthless."
The predicted class comes from the largest posterior, which depends only on rank order. Poor absolute calibration can coexist with correct ranking and good accuracy.
"Prior , but strong evidence should still make possible."
A prior of exactly 0 kills the numerator , so the posterior stays 0. Bayes can never resurrect a hypothesis you assigned zero prior — assign a tiny nonzero prior instead.

Why questions

Why does a positive test for a rare disease often leave you probably healthy?
The many healthy people (large ) generate a flood of false positives that outnumber the few true positives, so most positives come from healthy people.
Why do we bother with the denominator instead of just using the numerator?
The numerator is an unnormalized weight; dividing by rescales all hypotheses so their posteriors sum to 1 and stay .
Why can we ignore the denominator when comparing two classes?
is identical for both classes, so it cancels in the ratio ; only the numerators decide the winner.
Why does Bayes let us "reverse" a conditional probability?
Both and are built from the same joint . Bayes substitutes one expression for that joint into the other, swapping the direction of conditioning.
Why can yesterday's posterior become today's prior?
A posterior is just your updated belief; feeding it back as the prior for the next piece of evidence is exactly what "updating incrementally" means, and repeated application is consistent as long as the evidences are conditioned correctly.
Why does stronger evidence move beliefs more?
The bigger the gap between and , the more one hypothesis explains the data than the other, so the ratio of numerators — and hence the shift from prior to posterior — is larger.
Why does MLE (maximum likelihood) sometimes disagree with a Bayesian answer?
MLE maximizes only the likelihood and ignores the prior ; Bayes weights the likelihood by the prior, so with a strong prior the two can pick different hypotheses.

Edge cases

What is the posterior if the evidence is impossible, ?
Undefined — Bayes requires because you cannot condition on an event that never happens (division by zero).
What happens when the prior is ?
The posterior stays 1 regardless of evidence: you were already certain, so no observation can change a probability of exactly 1 (or exactly 0).
What is the posterior when the likelihood ?
It becomes 0. If makes impossible yet occurred, is ruled out completely — this is how evidence can eliminate a hypothesis.
In Naive Bayes, one feature's likelihood is exactly 0. What breaks?
The whole product collapses to 0, wrongly forbidding the class from ever winning. This is why smoothing (e.g. Laplace) adds a small count so no likelihood is exactly zero.
Two hypotheses have equal priors and equal likelihoods for the observed evidence. What are the posteriors?
They are equal — the evidence cannot distinguish them, so you leave with a 50/50 split (for two hypotheses) exactly as you started.
If the false-positive rate equals the true-positive rate, what does a positive test tell you?
Nothing about the diagnosis. With the posterior equals the prior — the test is diagnostically worthless despite producing "positive" results.
What is the posterior after seeing evidence that every hypothesis explains equally well?
It equals the prior for each hypothesis. Uniformly likely evidence is uninformative, so Bayes leaves your beliefs untouched — a useful sanity check.