Conditional probability

Overview

Conditional probability measures how the probability of an event changes when we learn that another event has occurred. It's the foundation of Bayesian reasoning, naive Bayes classifiers, Hidden Markov Models, and essentially all probabilistic inference in ML.

[!intuition] The Core Insight

When you gain information, you shrink the universe of possible outcomes. Conditional probability asks: "Given that I now know B happened, what fraction of those B-outcomes also include A?"

Physical analogy: You have 100 emails. 60 are spam, 40 are legitimate. Among spam, 50 contain the word "free". You open an email and see "free" — your probability estimate that it's spam updates because you've restricted your universe to only "free"-containing emails.

The formula $P(A|B)$ reads as "probability of A given B" — it's the probability that A occurs in the world where B is already true.

[!definition] Formal Definition

The conditional probability of event $A$ given event $B$ (where $P(B) > 0$) is:

$P(A|B) = \frac{P(A \cap B)}{P(B)}$

Why this formula?

• Numerator $P(A \cap B)$: The probability both A and B happen (the overlap).
• Denominator $P(B)$: Normalizes by treating B as the new "total universe" (renormalization).
• Result: The fraction of B-outcomes that also satisfy A.

Derivation from first principles (frequency interpretation):

1. Run $N$ trials. Event $B$ occurs in $N_B$ trials.
2. Among those $N_B$ trials, event $A$ occurs in $N_{AB}$ trials.
3. Empirical conditional probability: $\frac{N_{AB}}{N_B}$.
4. Express in probabilities: $\frac{N_{AB}/N}{N_B/N} = \frac{P(A \cap B)}{P(B)}$.

[!formula] Key Formulas & Derivations

1. Multiplication Rule (Chain Rule)

Rearranging the definition: $P(A \cap B) = P(A|B) \cdot P(B)$

Why? If you want both A and B to happen:

• First, B must happen: probability $P(B)$.
• Then, given B happened, A must also happen: probability $P(A|B)$.
• Multiply because these are sequential restrictions.

General chain rule (for multiple events): $P(A_1 \cap A_2 \cap \cdots \cap A_n) = P(A_1) \cdot P(A_2|A_1) \cdot P(A_3|A_1 \cap A_2) \cdots P(A_n|A_1 \cap \cdots \cap A_{n-1})$

Derivation: Apply multiplication rule recursively.

2. Law of Total Probability

If $B_1, B_2, \ldots, B_n$ partition the sample space (mutually exclusive, exhaustive): $P(A) = \sum_{i=1}^{n} P(A|B_i) \cdot P(B_i)$

Why? Event A can happen through any of the $B_i$ "pathways". Each pathway contributes $P(A|B_i) \cdot P(B_i)$ (probability of reaching that pathway times probability A happens there).

Derivation:

1. Since $B_i$ partition the space: $A = \bigcup_{i=1}^n (A \cap B_i)$ (disjoint union).
2. By additivity: $P(A) = \sum_{i=1}^n P(A \cap B_i)$.
3. By multiplication rule: $P(A \cap B_i) = P(A|B_i) \cdot P(B_i)$.
4. Combine: $P(A) = \sum_{i=1}^n P(A|B_i) \cdot P(B_i)$.

3. Bayes' Theorem

$P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}$

Derivation from multiplication rule:

• From definition: $P(A \cap B) = P(A|B) \cdot P(B)$.
• Also: $P(A \cap B) = P(B|A) \cdot P(A)$ (symmetry).
• Equate: $P(A|B) \cdot P(B) = P(B|A) \cdot P(A)$.
• Solve for $P(A|B)$: $P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}$.

Expanded form (using total probability for denominator): $P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B|A) \cdot P(A) + P(B|\neg A) \cdot P(\neg A)}$

Why this matters: You often know $P(B|A)$ (likelihood) and $P(A)$ (prior) but want $P(A|B)$ (posterior). Bayes flips the conditioning.

[!example] Worked Example 1: Medical Diagnosis

Setup: A disease affects 1% of the population. A test correctly identifies the disease 95% of the time (sensitivity) and correctly identifies healthy patients 90% of the time (specificity). You test positive. What's the probability you have the disease?

Define events:

• $D$: Has disease. $P(D) = 0.01$, $P(\neg D) = 0.99$.
• $T^+$: Tests positive.
• $P(T^+|D) = 0.95$ (sensitivity).
• $P(T^-|\neg D) = 0.90$ → $P(T^+|\neg D) = 0.10$ (false positive rate).

Want: $P(D|T^+)$.

Step 1: Apply Bayes' theorem. $P(D|T^+) = \frac{P(T^+|D) \cdot P(D)}{P(T^+)}$

Step 2: Calculate $P(T^+)$ using total probability (partition by $D$ and $\neg D$). $P(T^+) = P(T^+|D) \cdot P(D) + P(T^+|\neg D) \cdot P(\neg D)$ $= 0.95 \times 0.01 + 0.10 \times 0.99 = 0.0095 + 0.099 = 0.1085$

Why this step? We need to account for both ways to test positive: truly sick and falsely positive.

Step 3: Compute posterior. $P(D|T^+) = \frac{0.95 \times 0.01}{0.1085} = \frac{0.0095}{0.1085} \approx 0.0875 \approx 8.75\%$

Interpretation: Even with a positive test, you only have ~9% chance of disease because the disease is rare (low prior). Most positives are false positives from the healthy majority.

[!example] Worked Example 2: Spam Filter (Naive Bayes Foundation)

Setup:

• 60% of emails are spam, 40% are ham.
• Word "free" appears in 80% of spam, 10% of ham.
• An email contains "free". Is it spam?

Events:

• $S$: Spam. $P(S) = 0.6$, $P(H) = 0.4$ (ham).
• $F$: Contains "free".
• $P(F|S) = 0.8$, $P(F|H) = 0.1$.

Want: $P(S|F)$.

Step 1: Apply Bayes. $P(S|F) = \frac{P(F|S) \cdot P(S)}{P(F)}$

Step 2: Calculate $P(F)$. $P(F) = P(F|S) \cdot P(S) + P(F|H) \cdot P(H)$ $= 0.8 \times 0.6 + 0.1 \times 0.4 = 0.48 + 0.04 = 0.52$

Why this step? "Free" can appear in spam or ham; weight by their prior probabilities.

Step 3: Compute. $P(S|F) = \frac{0.8 \times 0.6}{0.52} = \frac{0.48}{0.52} \approx 0.923 \approx 92.3\%$

Interpretation: Seeing "free" strongly suggests spam because it's much more common in spam (80% vs 10%) and spam is already the majority class.

[!example] Worked Example 3: Independence Check

Setup: Two coin flips. $A$ = first is heads, $B$ = second is heads.

Check independence: Events are independent if $P(A|B) = P(A)$.

Calculate:

• $P(A) = 0.5$ (fair coin).
• $P(A \cap B) = 0.25$ (both heads).
• $P(B) = 0.5$.
• $P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{0.25}{0.5} = 0.5 = P(A)$.

Why this step? If learning B doesn't change the probability of A, they're independent. The second flip doesn't affect the first.

Alternative check: $P(A \cap B) = P(A) \cdot P(B)$? $0.25 = 0.5 \times 0.5 = 0.25 \checkmark$

Both checks confirm independence.

[!mistake] Common Mistakes

Mistake 1: Confusing $P(A|B)$ with $P(B|A)$

Wrong thinking: "If 90% of people with the disease test positive, then 90% of positive tests mean disease."

Why it feels right: The numbers sound similar; our brains conflate the two directions.

The fix: $P(A|B) \neq P(B|A)$ in general. They're related by Bayes' theorem but weighted by priors. In medical testing, $P(\text{test}+|\text{disease})$ (sensitivity) is NOT the same as $P(\text{disease}|\text{test}+)$ (posterior).

Steel-man: The confusion arises because in symmetric scenarios (like fair coins), $P(A|B) = P(B|A)$. But with asymmetric priors (rare disease, common test), they diverge wildly.

Mistake 2: Ignoring the Base Rate (Neglecting Priors)

Wrong thinking: "The test is 95% accurate, so a positive result means 95% chance of disease."

Why it feels right: We focus on the test's accuracy and ignore population prevalence.

The fix: Always include the prior $P(D)$ in Bayes' theorem. A 95% sensitive test on a 1% prevalence disease yields only ~9% posterior (Example 1).

Steel-man: Test accuracy is the most salient information presented, so our attention anchors there. The base rate is abstract and easy to forget.

Mistake 3: Forgetting to Normalize

Wrong thinking: "Both events have 50% probability conditional on B, so they're equally likely."

Why it feels right: We compare likelihoods directly without renormalizing.

The fix: $P(A|B)$ requires dividing by $P(B)$, the total probability of the conditioning event. Without normalization, you're not measuring fractions of the new universe.

Steel-man: In many contexts (like comparing hypotheses), ratios of unnormalized probabilities (likelihoods) suffice for ranking, so we skip normalization. But for actual probabilities, you must normalize.

[!recall]- Explain to a 12-Year-Old

Imagine you have a bag with 10 marbles: 6 red, 4 blue. Normally, the chance of grabbing red is 6/10 = 60%.

Now, your friend peeks and says, "I see it's a marble from the left half of the bag." Suppose the left half has 4 red and 1 blue. Now your universe shrunk to just those 5 marbles. The chance of red given it's from the left half is 4/5 = 80%. The "given left half" updated your probability because you have new information that changes what's possible.

Conditional probability is just: When you learn something new, recalculate chances in the smaller world where that new thing is true.

[!mnemonic] Remember It

"B is the New Total": When you see $P(A|B)$, imagine B is now 100% (the whole universe). A is a fraction of that. Divide the overlap by B's total.

"Bayes Flips the Pipe": $P(A|B)$ vs $P(B|A)$ — Bayes' theorem is the flipping tool. Prior → Likelihood → Posterior.

Properties of Conditional Probability

1. Range: $0 \leq P(A|B) \leq 1$ (still a probability).
2. Certainty: $P(B|B) = 1$ (if B happened, B definitely happened).
3. Additivity: $P(A \cup C|B) = P(A|B) + P(C|B)$ if $A \cap C = \emptyset$.
4. Independence: If $A$ and $B$ are independent, $P(A|B) = P(A)$ (learning B doesn't change A).

Connections

• Bayes-Theorem — Direct application and inversion of conditioning.
• Law-of-Total-Probability — Partitioning for marginalization.
• Independence — Special case where $P(A|B) = P(A)$.
• Naive-Bayes-Classifier — Uses $P(\text{class}|\text{features})$ via Bayes + independence assumptions.
• Hidden-Markov-Models — Transition probabilities are conditional: $P(s_t|s_{t-1})$.
• Bayesian-Inference — Updating beliefs: prior → likelihood → posterior.
• Mutual-Information — Measures how much learning Y reduces uncertainty about X.
• Chain-Rule-of-Probability — Factorizing joint distributions via conditionals.

Flashcards

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)