Multinomial and Bernoulli Naive Bayes

2.4.9 · AI-ML › SVM, Naive Bayes & Probabilistic Models

YEH sab kyun chahiye?

Hum chahte hain $P(\text{class}\mid \text{document})$. Bayes' theorem se:

$P(c\mid \mathbf{x})=\frac{P(c)P(\mathbf{x}\mid c)}{P(\mathbf{x})}$

Mushkil part hai $P(\mathbf{x}\mid c)$ — saare words par ek joint distribution. Yeh combinations astronomically zyada hain. "Naive" assumption humein bachata hai:

Ab hum ek waqt mein ek hi word estimate karte hain. Dono variants mein difference hai ki $P(x_i\mid c)$ kaisa dikhta hai.

Multinomial Naive Bayes

First principles se likelihood derive karna

Ek document hai $\mathbf{x}=(x_1,\dots ,x_{|V|})$ jahan $x_t$ = word $t$ ka count, aur $n={\sum }_t{x}_t$ length hai. $n$ baar die roll karna face-probabilities $p_{t\mid c}$ ke saath multinomial distribution deta hai:

$P(\mathbf{x}\mid c)=\frac{n!}{{\prod }_t{x}_t!}{\prod }_t{p}_{t\mid c}^{x_t}$

Yeh form kyun? ${\prod }_t{p}_{t\mid c}^{x_t}$ rolls ke ek specific ordering ki probability hai; $\frac{n!}{{\prod }_t{x}_t!}$ count karta hai kitni orderings same counts deti hain. Factorial part har class ke liye same hai, isliye classification mein cancel ho jaata hai aur hum usually ise drop kar dete hain.

Hum logs lete hain (products underflow karte hain, sums stable hote hain):

$\log P(c\mid \mathbf{x})\propto \log P(c)+{\sum }_t{x}_t\log p_{t\mid c}$

MLE derive kaise hoti hai (counts/total kyun?): maximize karo ${\sum }_t{N}_{tc}\log p_{t\mid c}$ subject to ${\sum }_t{p}_{t\mid c}=1$. Lagrangian $\mathcal{L}={\sum }_t{N}_{tc}\log p_t-\lambda ({\sum }_t{p}_t-1)$. $\partial \mathcal{L}/\partial p_t=N_{tc}/p_t-\lambda =0$ set karne par $p_t=N_{tc}/\lambda$ milta hai; constraint se $\lambda =N_c$ force hota hai, isliye $p_{t\mid c}=N_{tc}/N_c$. Smoothing bas isme thoda perturbation karta hai.

Bernoulli Naive Bayes

Likelihood derive karna

Features binary hain: $x_t\in \{0,1\}$ (present/absent). Har ek Bernoulli trial hai:

$P(x_t\mid c)=p_{t\mid c}^{x_t}(1-p_{t\mid c}{)}^{1-x_t}$

Yeh trick kyun? Jab $x_t=1$ hota hai toh $p_{t\mid c}$ select hota hai; jab $x_t=0$ hota hai toh $(1-p_{t\mid c})$ select hota hai. Ek formula, do cases. Saare words par:

$\log P(c\mid \mathbf{x})\propto \log P(c)+{\sum }_{t\in V}[x_t\log p_{t\mid c}+(1-x_t)\log (1-p_{t\mid c})]$

Worked Example (dono models same data par)

Vocabulary $V=\{\text{cheap},\text{meeting},\text{free}\}$, $|V|=3$.

Spam docs: D1 = "cheap cheap free", D2 = "free free cheap" Ham docs: D3 = "meeting free", D4 = "meeting meeting"

Priors: $P(\text{spam})=P(\text{ham})=1/2$.

Multinomial ($\alpha =1$)

Spam mein counts: cheap=3, free=3, meeting=0, isliye $N_{\text{spam}}=6$. $p_{\text{cheap}\mid s}=\frac{3+1}{6+3}=\frac{4}{9},p_{\text{free}\mid s}=\frac{4}{9},p_{\text{meeting}\mid s}=\frac{1}{9}$ Ham mein counts: meeting=3, free=1, cheap=0, $N_{\text{ham}}=4$. $p_{\text{meeting}\mid h}=\frac{3+1}{4+3}=\frac{4}{7},p_{\text{free}\mid h}=\frac{2}{7},p_{\text{cheap}\mid h}=\frac{1}{7}$

"cheap free" classify karo (counts: cheap=1, free=1, meeting=0):

• Spam score $\propto \frac{1}{2}\cdot p_{\text{cheap}\mid s}{p}_{\text{free}\mid s}=\frac{1}{2}\cdot \frac{4}{9}\cdot \frac{4}{9}=\frac{16}{162}=0.0988$
• Ham score $\propto \frac{1}{2}\cdot \frac{1}{7}\cdot \frac{2}{7}=\frac{2}{98}=0.0204$

Yeh step kyun? Sirf present words (nonzero count) contribute karte hain; "meeting" ka count 0 hai isliye $p^0=1$ aur woh drop ho jaata hai. → SPAM.

Bernoulli ($\alpha =1$)

Document-presence counts, spam ($D_s=2$): cheap {D1,D2} mein=2, free=2, meeting=0. $p_{\text{cheap}\mid s}=\frac{2+1}{2+2}=\frac{3}{4},p_{\text{free}\mid s}=\frac{3}{4},p_{\text{meeting}\mid s}=\frac{1}{4}$ Ham ($D_h=2$): meeting=2, free=1, cheap=0. $p_{\text{meeting}\mid h}=\frac{3}{4},p_{\text{free}\mid h}=\frac{2}{4}=\frac{1}{2},p_{\text{cheap}\mid h}=\frac{1}{4}$

"cheap free" classify karo → binary vector cheap=1, free=1, meeting=0:

• Spam $\propto \frac{1}{2}\cdot p_{\text{cheap}}\cdot p_{\text{free}}\cdot (1-p_{\text{meeting}})=\frac{1}{2}\cdot \frac{3}{4}\cdot \frac{3}{4}\cdot \frac{3}{4}=0.2109$
• Ham $\propto \frac{1}{2}\cdot \frac{1}{4}\cdot \frac{1}{2}\cdot (1-\frac{3}{4})=\frac{1}{2}\cdot \frac{1}{4}\cdot \frac{1}{2}\cdot \frac{1}{4}=\frac{1}{64}=0.0156$

Yeh step kyun? Dekho extra $(1-p_{\text{meeting}})$ factor — Bernoulli "meeting" ki absence ko reward karta hai (jo ek hammy word hai). Multinomial ne ise bilkul ignore kiya tha. → SPAM (aur bhi zyada confidence se).

Flashcards

Connections

• Bayes Theorem — underlying inference rule jo dono models apply karte hain.
• Naive Bayes Classifier — general framework; yeh dono likelihood choices hain.
• Gaussian Naive Bayes — continuous-feature cousin ($P(x\mid c)$ ek normal density hai).
• Laplace Smoothing — regularization jo dono estimates mein use hoti hai.
• Bag of Words / TF-IDF — feature representations jo in models ko feed karte hain.
• Log-Sum-Exp trick — log-scores ka numerically stable normalization.
• Maximum Likelihood Estimation — parameter formulas ka source.

Concept Map