Bayes' theorem and applications: Exercises — AI-ML deep dive

Definition The four quantities in every Bayes problem

Say we care about a hypothesis $H$ (e.g. "has the disease") and we observe evidence $E$ (e.g. "test is positive").

$P (H)$ = the prior — how much we believed $H$ before looking at any evidence. A plain fraction of the population.
$P (E ∣ H)$ = the likelihood — "if $H$ were true, how often would we see this evidence?" Read the bar $∣$ as the word "given".
$P (E ∣ \neg H)$ = the false-alarm likelihood — how often the evidence shows up when $H$ is false. The symbol $\neg H$ means "not $H$ ".
$P (H ∣ E)$ = the posterior — our updated belief in $H$ after seeing the evidence. This is the answer we hunt for.

The engine that connects them (built from scratch in the parent): $P (H ∣ E) = \frac{P ( E ∣ H ) P ( H )}{P ( E ∣ H ) P ( H ) + P ( E ∣ \neg H ) P ( \neg H )}$ The bottom is just every way the evidence could appear — via $H$ or via $\neg H$ — which is the law of total probability.

A picture of what the formula is doing — it splits a population into four boxes and asks "of the two positive boxes, what fraction is the true one?"

Figure — Bayes' theorem and applications

Level 1 — Recognition

(Can you name the pieces and plug in? No traps yet — just fluency.)

Recall Solution 1.1

$P (H) = 0.03$ — the prior, the base rate of defective chips.
$P (E ∣ H) = 0.90$ — the likelihood: flagged given truly defective.
$P (E ∣ \neg H) = 0.04$ — the false-alarm rate: flagged given good. Also $P (\neg H) = 1 - 0.03 = 0.97$ . That is all Level 1 asks: matching English sentences to symbols.

Recall Solution 1.2

$P (H ∣ E)$ — the probability the chip is actually defective given that the scanner flagged it. This is the posterior. The manager does not care how often the scanner fires on bad chips ( $P (E ∣ H)$ ); they care whether this flagged chip is truly bad.

Level 2 — Application

(Now run the full machine and get a number.)

Recall Solution 2.1

WHAT we do: plug into the expanded Bayes formula. WHY: we are never told $P (E)$ directly, so we build it from the two ways a chip can be flagged. $P (H ∣ E) = \frac{0.90 \times 0.03}{0.90 \times 0.03 + 0.04 \times 0.97}$ Top (true positives): $0.90 \times 0.03 = 0.027$ . False positives: $0.04 \times 0.97 = 0.0388$ . $P (H ∣ E) = \frac{0.027}{0.027 + 0.0388} = \frac{0.027}{0.0658} \approx 0.4103$ Answer: about 41%. Even a fairly sharp scanner leaves real doubt, because good chips vastly outnumber bad ones, so their 4% false alarms pile up.

Recall Solution 2.2

$H$ = spam, $E$ = contains "lottery". $P (H) = 0.30$ , $P (E ∣ H) = 0.60$ , $P (E ∣ \neg H) = 0.02$ , $P (\neg H) = 0.70$ . $P (H ∣ E) = \frac{0.60 \times 0.30}{0.60 \times 0.30 + 0.02 \times 0.70} = \frac{0.18}{0.18 + 0.014} = \frac{0.18}{0.194} \approx 0.9278$ Answer: about 92.8%. A word that is 30× rarer in ham than spam ( $0.60$ vs $0.02$ ) is powerful evidence. See 2.1.05-Naive-Bayes-classifier for how many such words get multiplied together.

Recall Solution 2.3

WHY odds form? It skips the denominator entirely — handy when you only need to compare, or when chaining evidence.

Prior odds $= \frac{P ( H )}{P ( \neg H )} = \frac{0.30}{0.70} = \frac{3}{7}$ .
Likelihood ratio $= \frac{P ( E ∣ H )}{P ( E ∣ \neg H )} = \frac{0.60}{0.02} = 30$ .
Posterior odds $= \frac{3}{7} \times 30 = \frac{90}{7} \approx 12.857$ . Convert odds $o$ back to probability with $p = \frac{o}{1 + o}$ : $p = \frac{12.857}{13.857} \approx 0.9278.$ Same 92.8% — the two roads meet.

Level 3 — Analysis

(Explain why the number behaves as it does; probe sensitivity.)

Recall Solution 3.1

WHAT we do: Bayesian updating — the posterior after test 1 becomes the prior for test 2. WHY: independent evidence multiplies; the odds form makes this clean.

Prior odds $= \frac{0.03}{0.97} \approx 0.030928$ .
Likelihood ratio per positive scan $= \frac{0.90}{0.04} = 22.5$ .
Two positive scans: multiply the ratio twice. $posterior odds = 0.030928 \times 22.5 \times 22.5 = 0.030928 \times 506.25 \approx 15.657$ $p = \frac{15.657}{1 + 15.657} \approx 0.9400$ Answer: about 94.0%. One flag left us at 41%; a second independent flag rockets us to 94%. WHY: each 22.5× likelihood ratio is strong, and stacking two overpowers the rare prior. This is exactly point 3 ("update incrementally") in the parent's "Why Bayes is powerful."

Recall Solution 3.2

Setup: posterior $= 0.5$ means true-positive mass equals false-positive mass. $0.95 p = 0.05 (1 - p)$ WHY this equation: at 50% posterior the two boxes in the denominator are equal, so their numerators must match. $0.95 p = 0.05 - 0.05 p \Rightarrow 1.0 p = 0.05 \Rightarrow p = 0.05.$ Answer: a 5% base rate. Below 5% prevalence, a single positive is more likely wrong than right; above it, more likely right. This one number explains the whole "base-rate fallacy": the test's usefulness hinges on how common the disease already is.

Level 4 — Synthesis

(Combine several tools: total probability, multiple hypotheses, and independence.)

Recall Solution 4.1

WHAT: three mutually exclusive hypotheses, so use the extended Bayes (denominator = sum over all suppliers). WHY: the bulb came from exactly one supplier; they partition the sample space. Numerators $P (def ∣ X) P (X)$ :

A: $0.01 \times 0.50 = 0.005$
B: $0.02 \times 0.30 = 0.006$
C: $0.05 \times 0.20 = 0.010$ Total (evidence) $= 0.005 + 0.006 + 0.010 = 0.021$ . Posteriors: $P (A ∣ def) = \frac{0.005}{0.021} \approx 0.2381, P (B ∣ def) = \frac{0.006}{0.021} \approx 0.2857, P (C ∣ def) = \frac{0.010}{0.021} \approx 0.4762.$ Answer: supplier C, with ≈ 47.6%. Though C makes only 20% of bulbs, its high defect rate makes it the top suspect. Notice the three posteriors sum to 1 — the denominator's job.

The bar chart below shows how the small supplier C wins the "defective" verdict despite the smallest share of production:

Recall Solution 4.2

WHY multiply: the naive assumption says both words' likelihoods multiply within each class.

Spam numerator: $0.4 \times 0.8 \times 0.5 = 0.16$ .
Ham numerator: $0.6 \times 0.1 \times 0.05 = 0.003$ . $P (spam ∣ both) = \frac{0.16}{0.16 + 0.003} = \frac{0.16}{0.163} \approx 0.9816$ Answer: about 98.2%. Two independent spammy words compound: each multiplies the odds, so evidence stacks fast — the same mechanism as repeated tests in Exercise 3.1.

Level 5 — Mastery

(Invent, defend, and reason about the model itself.)

Recall Solution 5.1

A test carries no information when a positive is equally likely under $H$ and $\neg H$ : $P (E ∣ H) = P (E ∣ \neg H) .$ Then the likelihood ratio is $1$ , posterior odds = prior odds, and the posterior equals the prior — the test taught us nothing. If instead $P (E ∣ H) < P (E ∣ \neg H)$ , the likelihood ratio is below 1, and a positive result lowers your belief. Example: sensitivity $P (E ∣ H) = 0.99$ but $P (E ∣ \neg H) = 0.999$ gives ratio $0.99/0.999 < 1$ , so a "positive" is actually mild evidence against disease. Lesson: high sensitivity alone means nothing; the ratio of the two likelihoods is what moves belief.

Recall Solution 5.2

Let prior odds $= o_{0} > 0$ and $r > 1$ . After $k$ tests, odds $= o_{0} r^{k}$ . Since $r > 1$ , $r^{k} \to \infty$ as $k \to \infty$ , so odds $\to \infty$ , and $p = \frac{o _{0} r ^{k}}{1 + o _{0} r ^{k}} \to 1$ . So in principle enough independent positives overwhelm any prior. The catch: real repeated tests are not independent — the same underlying flaw (a cross-reacting antibody, a scanner biased by chip material) makes a false positive recur, so $r$ collapses toward 1 on repeats. The mathematics is sound; the independence assumption is the fragile part — the very lesson of Mistake 3 on treating independence as truth. Compare with the frequentist stance in 3.4.01-Maximum-likelihood-estimation, which drops priors entirely, and the fuller machinery in 4.2.03-Bayesian-inference.