Bayes' theorem and applications

What is Bayes' Theorem?

WHY does this formula work? Let's derive it from scratch.

Derivation from First Principles

Start with the definition of conditional probability: $P(A|B) = \frac{P(A \cap B)}{P(B)}$

WHY? This says: "probability of $A$ given $B$ " = "both happen together" divided by "how often $B$ happens". If $B$ is rare, seeing $A$ and $B$ together is relatively more significant.

Similarly, we can write: $P(B|A) = \frac{P(A \cap B)}{P(A)}$

Rearrange this second equation: $P(A \cap B) = P(B|A) \cdot P(A)$

WHY? We're expressing the joint probability in terms of "how likely is $B$ if we know $A$ " times "how likely is $A$ in the first place".

Substitute into the first equation: $P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}$

This is Bayes' theorem. It's just algebra on conditional probability definitions. The beauty: it reverses conditioning. If you know $P(B|A)$ but want $P(A|B)$ , Bayes gives you the swap.

Expanding the Denominator

Often $P(B)$ isn't directly known. Use the law of total probability: $P(B) = P(B|A) \cdot P(A) + P(B|\neg A) \cdot P(\neg A)$

WHY? Event $B$ can happen in two mutually exclusive scenarios: when $A$ is true or when $A$ is false. Sum the weighted probabilities.

So Bayes becomes: $P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B|A) \cdot P(A) + P(B|\neg A) \cdot P(\neg A)}$

This form is most useful in practice because you can compute the posterior from likelihoods and priors alone.

Figure — Bayes' theorem and applications

Worked Examples

Worked example Example 1: Medical Diagnosis (Classic Application)

A disease affects 1% of the population. A test is95% accurate (true positive rate = 0.95, false positive rate = 0.05). Question: If you test positive, what's the probability you actually have the disease? Solution:

Let $D$ = have disease, $T$ = test positive
Prior: $P(D) = 0.01$ , $P(\neg D) = 0.99$
Likelihood: $P(T|D) = 0.95$ (sensitivity), $P(T|\neg D) = 0.05$ (false positive rate) Apply Bayes: $P(D|T) = \frac{P(T|D) \cdot P(D)}{P(T|D) \cdot P(D) + P(T|\neg D) \cdot P(\neg D)}$ WHY this step? We're using the extended form because we don't know $P(T)$ directly, but we can compute it from the two scenarios.

$P(D|T) = \frac{0.95 \times 0.01}{0.95 \times 0.01 + 0.05 \times 0.99}$ $= \frac{0.0095}{0.0095 + 0.0495} = \frac{0.095}{0.059} \approx 0.161$

Answer: Only 16.1% chance you have the disease!

WHY so low? The disease is rare (low prior). Even though the test is 95% accurate, there are many more healthy people (99%) who can trigger the 5% false positive. The false positives outnumber true positives.

Worked example Example 2: Email Spam Classification

You receive an email with the word "FREE".

40% of all emails are spam (prior)
80% of spam emails contain "FREE" (likelihood)
10% of non-spam emails contain "FREE"

Question: What's the probability this email is spam? Solution:

Let $S$ = spam, $F$ = contains "FREE"
$P(S) = 0.4$ , $P(\neg S) = 0.6$
$P(F|S) = 0.8$ , $P(F|\neg S) = 0.1$

$P(S|F) = \frac{P(F|S) \cdot P(S)}{P(F|S) \cdot P(S) + P(F|\neg S) \cdot P(\neg S)}$

WHY? Same structure: posterior = (likelihood × prior) / evidence.

$P(S|F) = \frac{0.8 \times 0.4}{0.8 \times 0.4 + 0.1 \times 0.6} = \frac{0.32}{0.32 + 0.06} = \frac{0.32}{0.38} \approx 0.842$

Answer: 84.2% chance it's spam.

Insight: The word "FREE" is 8× more likely in spam than non-spam (0.8 vs 0.1), which strongly updates our belief.

Worked example Example 3: Machine Learning - Naive Bayes Classifier

Train a classifier on features $x_1, x_2, \ldots, x_n$ for class $C$ .

Goal: Compute $P(C|x_1, x_2, \ldots, x_n)$

Naive assumption: Features are independent given the class. $P(x_1, x_2, \ldots, x_n | C) = P(x_1|C) \cdot P(x_2|C) \cdots P(x_n|C)$

WHY naive? Real features often correlate, but independence simplifies computation massively.

Apply Bayes: $P(C|x_1, \ldots, x_n) = \frac{P(C) \cdot \prod_{i=1}^n P(x_i|C)}{P(x_1, \ldots, x_n)}$ In practice: Compare $P(C_1|\mathbf{x})$ vs $P(C_2|\mathbf{x})$ . The denominator cancels in the ratio, so: $\frac{P(C_1|\mathbf{x})}{P(C_2|\mathbf{x})} = \frac{P(C_1) \prod P(x_i|C_1)}{P(C_2) \prod P(x_i|C_2)}$ Pick the class with higher posterior.

Why Bayes is Powerful in AI/ML

Probabilistic reasoning: Handles uncertainty explicitly (vs deterministic rules)
Incorporates prior knowledge: If you know disease is rare, prior $P(D)$ is low → harder to confirm
Updates incrementally: See more evidence? Apply Bayes again with the old posterior as the new prior
Foundation for algorithms: Naive Bayes,esian networks, spam filters, medical AI A/B testing

Common Mistakes

Connections

1.3.01-Conditional-probability-and-independence – Bayes is built on conditional probability
1.3.02-Law-of-total-probability – Used to expand $P(B)$ in the denominator
2.1.05-Naive-Bayes-classifier – Direct ML application
4.2.03-Bayesian-inference – Philosophical framework for updating beliefs
3.4.01-Maximum-likelihood-estimation – Frequentist alternative; MLE ignores priors
1.3.04-Random-variables-and-distributions – Bayes extends to continuous distributions (Bayesian updating)

Recall Explain to a 12-Year-Old

Imagine you're a detective. Someone stole cookies, and you find chocolate crumbs. Your suspects: your brother (who loves chocolate, steals cookies 80% of the time when he's around) and your dog (who eats anything, but only20% chance he'd leave crumbs).

Before seeing the crumbs, you thought your brother was around 40% of the time (that's your prior). The crumbs are evidence. Your brother would definitely leave chocolate crumbs if he stole them (high likelihood). The dog might, but less often.

Bayes' theorem is the math that tells you: "Okay, you saw chocolate crumbs. Given that, what's the NEW chance it was your brother?" It combines your starting guess with how well each suspect explains the crumbs. The one who's more likely to create that evidence gets a bigger update. That's Bayes!

#flashcards/ai-ml

What is the formula for Bayes' theorem? :: $P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}$ where $P(A|B)$ is posterior, $P(B|A)$ is likelihood, $P(A)$ is prior, $P(B)$ is evidence.

What are the four components of Bayes' theorem? :: Posterior $P(A|B)$ , Likelihood $P(B|A)$ , Prior $P(A)$ , Evidence/Marginal Likelihood $P(B)$ .

Why is the denominator in Bayes' theorem necessary?

It normalizes the posterior probability so that all posteriors sum to 1 and prevents probabilities from exceeding 1.

How do you expand

P(B)

when it's not directly known?

Use the law of total probability:

P(B) = P(B|A) \cdot P(A) + P(B|\neg A) \cdot P(\neg A)

In the medical test example, why is the posterior probability only 16% despite95% test accuracy?

Because the disease is rare (1% prior). The false positives from the large healthy population (99% × 5% = 4.95%) outnumber the true positives (1% × 95% = 0.95%).

What does "naive" mean in Naive Bayes classifier?

It assumes features are conditionally independent given the class, which is usually false but simplifies computation and often works well in practice.

What is the difference between

P(A|B)

and

P(B|A)

P(A|B)

is the probability of

A

given

B

occurred (posterior), while

P(B|A)

is the probability of

B

given

A

occurred (likelihood). They are generally not equal.

How does Bayes' theorem handle multiple hypotheses?

P(A_i|B) = \frac{P(B|A_i) \cdot P(A_i)}{\sum_{j=1}^n P(B|A_j) \cdot P(A_j)}

where hypotheses

A_i

are mutually exclusive and exhaustive.

What is the role of the prior probability in Bayes' theorem?

It represents initial belief before seeing evidence. A strong prior (very high or very low) requires strong evidence to shift significantly.

Why does Naive Bayes work despite the independence assumption being violated?

Because the rank order of posterior probabilities is often correct even when absolute values are miscalibrated, which is sufficient for classification.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Bayes' theorem ek aisa powerful formula hai jo apko bata hai ki naye evidence ke baad apni belief ko kaise update karein. Socho ek doctor ko – initially usko lagta hai 1% chance hai ki patient ko disease hai (ye hai prior). Phir test positive ata hai. Ab doctor Bayes' theorem use karta hai: test kitna accurate hai (likelihood), disease kitna common hai (prior), aur test positive ane ki total probability (evidence) – sab combine karke posterior nikalta hai, matlab updated belief. Interesting baat ye hai ki agar disease bohot rare hai, toh bhi 95% accurate test positive aye, toh bhi actual disease hone ki probability sirf 16% ho sakti hai! Kyunki false positives (healthy log jinko test galat positive bata deta hai) true positives se zyada hote hain.

AI aur machine learning mein Bayes ka application bohot common hai. Naive Bayes classifier spam detection, sentiment analysis jaise tasks mein use hota hai. Isme aap assume karte ho ki features independent hain (jo usually galat assumption hai, isliye "naive" naam hai), lekin phir bhi practically bohot acha kaam karta hai. Bayes ka core idea simple hai: prior belief + new evidence = updated belief. Medical diagnosis se lekar email filtering tak, jahan bhi uncertainty hai aur apko probabilities ke sath kaam karna hai, Bayes theorem aapka fundamental tool hai. Isko samajhna AI ke liye bohot zaroori hai kyunki ye uncertainty ko mathematically handle karne ka clean tarika deta hai.

What is Bayes' Theorem?

Derivation from First Principles

Expanding the Denominator

Worked Examples

Why Bayes is Powerful in AI/ML

Common Mistakes

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability & Statistics

Connections