Naive Bayes assumption — AI-ML notes

WHY do we need an assumption at all?

We want to classify: given a feature vector $\mathbf{x} = (x_1, x_2, \dots, x_n)$ , pick the class $y$ that is most probable.

The troublemaker is the likelihood $P(\mathbf{x}\mid y) = P(x_1, x_2, \dots, x_n \mid y)$ .

The chain rule — before we get naive

By repeated use of $P(A,B)=P(A\mid B)P(B)$ : $P(x_1,\dots,x_n\mid y)=P(x_1\mid y)\,P(x_2\mid x_1,y)\,P(x_3\mid x_1,x_2,y)\cdots$

This is exact but useless — the later terms still depend on all earlier features. Now we make the leap.

Each $P(x_i\mid y)$ is a simple 1-D distribution — total parameters drop from $k^n$ to about $n\cdot k$ . That is the payoff.

HOW the full classifier is built

Combine Bayes' rule with the factorization: $P(y\mid \mathbf{x}) = \frac{P(y)\prod_{i=1}^{n}P(x_i\mid y)}{P(\mathbf{x})}$

Worked Example 1 — Spam filter (Bernoulli/multinomial words)

Vocabulary features. $P(\text{spam})=0.4$ , $P(\text{ham})=0.6$ . Word likelihoods:

word	$P(w\mid\text{spam})$	$P(w\mid\text{ham})$
"free"	0.7	0.1
"win"	0.5	0.05

Message contains both "free" and "win". Score each class:

Spam: $0.4 \times 0.7 \times 0.5 = 0.14$ Why this step? Prior $\times$ product of independent word likelihoods (naive assumption).
Ham: $0.6 \times 0.1 \times 0.05 = 0.003$

$0.14 > 0.003 \Rightarrow$ predict spam. Normalized: $P(\text{spam}\mid\mathbf x)=\frac{0.14}{0.143}\approx 0.979$ . Why normalize? To turn the raw scores into a probability we divide by their sum $=P(\mathbf x)$ .

Worked Example 2 — Gaussian Naive Bayes (continuous features)

Feature = height, two classes. For continuous $x_i$ we replace the table with a Gaussian likelihood: $P(x_i\mid y)=\frac{1}{\sqrt{2\pi\sigma_y^2}}\exp\!\Big(-\frac{(x_i-\mu_y)^2}{2\sigma_y^2}\Big)$

Say class A: $\mu=170,\sigma=10$ ; class B: $\mu=185,\sigma=8$ ; equal priors. New point $x=178$ .

$\log P(x\mid A) = -\tfrac12\log(2\pi\cdot 100) - \frac{(178-170)^2}{2\cdot100} = -\tfrac12\log(628.3)-0.32$
$\log P(x\mid B) = -\tfrac12\log(2\pi\cdot 64) - \frac{(178-185)^2}{2\cdot64} = -\tfrac12\log(402.1)-0.383$

Why logs? Adds numerical stability and lets us compare sums. Class B's smaller variance + closer mean wins here → predict B. (The narrower Gaussian rewards being close to its mean.)

Steel-manning the mistakes

Recall Feynman: explain to a 12-year-old

Imagine sorting fruit into "apple" or "orange". Once I tell you it's an orange, you can guess its color and its bumpy skin separately without them needing to consult each other — knowing it's an orange already explains why they go together. Naive Bayes pretends every clue is like that: once you know the answer-box, each clue speaks on its own. Then you just multiply how well each clue fits each box and pick the biggest.

Flashcards

What is the Naive Bayes assumption?

Given the class label, all features are conditionally independent:

P(x_i\mid x_{<i},y)=P(x_i\mid y)

How does the assumption simplify the likelihood?

It turns the joint

P(x_1,\dots,x_n\mid y)

into a product

\prod_i P(x_i\mid y)

State the Naive Bayes decision rule.

\hat y=\arg\max_y P(y)\prod_i P(x_i\mid y)

Why can

P(\mathbf x)

be dropped from the argmax?

It's a constant across classes, so it doesn't change which class is maximal.

Why do we take logarithms in Naive Bayes?

To avoid numerical underflow from multiplying many small probabilities; log is monotonic so argmax is unchanged.

Difference between conditional and marginal independence?

Conditional independence holds given

y

; features may still be dependent marginally (overall).

What breaks if a feature has zero conditional probability?

The entire product becomes zero, killing that class regardless of other evidence.

What fixes zero-probability features?

Laplace/add-one smoothing:

\frac{\text{count}+1}{\text{total}+|V|}

Why does Naive Bayes work despite a wrong assumption?

Classification only needs the correct argmax (ranking), not calibrated probabilities.

Gaussian NB likelihood for continuous

x_i

\frac{1}{\sqrt{2\pi\sigma_y^2}}\exp(-(x_i-\mu_y)^2/2\sigma_y^2)

Connections

Bayes' Theorem — the exact rule NB approximates the likelihood inside.
Chain Rule of Probability — what the assumption is applied to.
Conditional Independence — the formal property assumed.
Laplace Smoothing — fixes zero-probability blow-ups.
Gaussian Naive Bayes / Multinomial Naive Bayes — variants by feature type.
Maximum A Posteriori (MAP) — NB is MAP with the product likelihood.
Logistic Regression — discriminative counterpart to generative NB.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Naive Bayes ka core idea bahut simple hai: hum class predict karna chahte hain given features $\mathbf{x}$ . Bayes rule se $P(y\mid\mathbf x)\propto P(\mathbf x\mid y)P(y)$ . Problem yeh hai ki poora joint $P(x_1,\dots,x_n\mid y)$ estimate karna impossible hai — bahut saare features ke saare combinations ke liye data hi nahi hoga. Isliye hum ek "naive" (bhola) assumption maarte hain: agar class pata ho, to saare features ek dusre se independent ho jaate hain. Matlab features aapas mein gossip karte hain, lekin class label unhe chup kara deta hai.

Is assumption se magic hota hai: joint distribution ek simple product ban jaata hai — $P(\mathbf x\mid y)=\prod_i P(x_i\mid y)$ . Ab har feature ka sirf apna chhota 1-D distribution estimate karo. Final rule: $\hat y=\arg\max_y P(y)\prod_i P(x_i\mid y)$ . $P(\mathbf x)$ ko hata dete hain kyunki wo har class ke liye same hai, ranking nahi badalti. Aur multiply karne se numbers bahut chhote ho jaate hain, isliye log le lete hain — sum ban jaata hai aur underflow nahi hota.

Do important gotchas yaad rakho. Pehla: agar koi word spam mein kabhi dikha hi nahi, to uska probability 0 ho jaata hai, aur ek 0 poore product ko 0 kar deta hai — isliye Laplace smoothing (add-one) lagate hain. Doosra: assumption real data mein aksar galat hoti hai (features correlated hote hain), phir bhi Naive Bayes accha classify karta hai, kyunki hume sirf sahi argmax chahiye, exact probability nahi. Isiliye spam filters mein yeh aaj bhi kaam karta hai — fast, simple, aur surprisingly strong baseline.

Test yourself — SVM, Naive Bayes & Probabilistic Models

Connections