Multinomial and Bernoulli Naive Bayes

WHY do we need these at all?

We want $P(\text{class} \mid \text{document})$ . By Bayes' theorem:

$P(c \mid \mathbf{x}) = \frac{P(c)\,P(\mathbf{x}\mid c)}{P(\mathbf{x})}$

The hard part is $P(\mathbf{x}\mid c)$ — a joint distribution over all words. That's astronomically many combinations. The "naive" assumption rescues us:

Now we only estimate one word at a time. The two variants differ in what $P(x_i\mid c)$ looks like.

A document is $\mathbf{x} = (x_1,\dots,x_{|V|})$ where $x_t$ = count of word $t$ , and $n=\sum_t x_t$ is the length. Rolling a die $n$ times with face-probabilities $p_{t\mid c}$ gives the multinomial distribution:

$P(\mathbf{x}\mid c) = \frac{n!}{\prod_t x_t!}\prod_t p_{t\mid c}^{\,x_t}$

Why this form? $\prod_t p_{t\mid c}^{x_t}$ is the probability of one specific ordering of the rolls; $\frac{n!}{\prod_t x_t!}$ counts how many orderings give the same counts. The factorial part is the same for every class, so it cancels in classification and we usually drop it.

We take logs (products underflow, sums are stable):

$\log P(c\mid \mathbf{x}) \;\propto\; \log P(c) + \sum_t x_t \,\log p_{t\mid c}$

HOW we derive the MLE (why counts/total?): maximize $\sum_t N_{tc}\log p_{t\mid c}$ subject to $\sum_t p_{t\mid c}=1$ . Lagrangian $\mathcal{L}=\sum_t N_{tc}\log p_{t} - \lambda(\sum_t p_t -1)$ . Setting $\partial\mathcal{L}/\partial p_t = N_{tc}/p_t - \lambda = 0$ gives $p_t = N_{tc}/\lambda$ ; the constraint forces $\lambda=N_c$ , so $p_{t\mid c}=N_{tc}/N_c$ . Smoothing just perturbs this.

Bernoulli Naive Bayes

Deriving the likelihood

Features are binary: $x_t\in\{0,1\}$ (present/absent). Each is a Bernoulli trial:

$P(x_t\mid c) = p_{t\mid c}^{\,x_t}\,(1-p_{t\mid c})^{\,1-x_t}$

Why this trick? When $x_t=1$ it selects $p_{t\mid c}$ ; when $x_t=0$ it selects $(1-p_{t\mid c})$ . One formula, two cases. Over all words:

$\log P(c\mid \mathbf{x}) \propto \log P(c) + \sum_{t\in V}\Big[x_t\log p_{t\mid c} + (1-x_t)\log(1-p_{t\mid c})\Big]$

Figure — Multinomial and Bernoulli Naive Bayes

Worked Example (both models on the same data)

Vocabulary $V=\{\text{cheap},\ \text{meeting},\ \text{free}\}$ , $|V|=3$ .

Spam docs: D1 = "cheap cheap free", D2 = "free free cheap" Ham docs: D3 = "meeting free", D4 = "meeting meeting"

Priors: $P(\text{spam})=P(\text{ham})=1/2$ .

Multinomial ( $\alpha=1$ )

Counts in spam: cheap=3, free=3, meeting=0, so $N_{\text{spam}}=6$ . $p_{\text{cheap}\mid s}=\frac{3+1}{6+3}=\tfrac49,\quad p_{\text{free}\mid s}=\tfrac49,\quad p_{\text{meeting}\mid s}=\tfrac19$ Counts in ham: meeting=3, free=1, cheap=0, $N_{\text{ham}}=4$ . $p_{\text{meeting}\mid h}=\frac{3+1}{4+3}=\tfrac47,\ p_{\text{free}\mid h}=\tfrac27,\ p_{\text{cheap}\mid h}=\tfrac17$

Classify "cheap free" (counts: cheap=1, free=1, meeting=0):

Spam score $\propto \tfrac12 \cdot p_{\text{cheap}\mid s}\,p_{\text{free}\mid s} = \tfrac12\cdot\tfrac49\cdot\tfrac49 = \tfrac{16}{162}=0.0988$
Ham score $\propto \tfrac12\cdot\tfrac17\cdot\tfrac27 = \tfrac{2}{98}=0.0204$

Why this step? Only words present (nonzero count) contribute; "meeting" has count 0 so $p^0=1$ and drops out. → SPAM.

Bernoulli ( $\alpha=1$ )

Document-presence counts, spam ( $D_s=2$ ): cheap in {D1,D2}=2, free=2, meeting=0. $p_{\text{cheap}\mid s}=\frac{2+1}{2+2}=\tfrac34,\ p_{\text{free}\mid s}=\tfrac34,\ p_{\text{meeting}\mid s}=\tfrac14$ Ham ( $D_h=2$ ): meeting=2, free=1, cheap=0. $p_{\text{meeting}\mid h}=\tfrac34,\ p_{\text{free}\mid h}=\tfrac24=\tfrac12,\ p_{\text{cheap}\mid h}=\tfrac14$

Classify "cheap free" → binary vector cheap=1, free=1, meeting=0:

Spam $\propto \tfrac12\cdot p_{\text{cheap}}\cdot p_{\text{free}}\cdot(1-p_{\text{meeting}}) = \tfrac12\cdot\tfrac34\cdot\tfrac34\cdot\tfrac34 = 0.2109$
Ham $\propto \tfrac12\cdot\tfrac14\cdot\tfrac12\cdot(1-\tfrac34)=\tfrac12\cdot\tfrac14\cdot\tfrac12\cdot\tfrac14 = \tfrac{1}{64}=0.0156$

Why this step? Notice the extra $(1-p_{\text{meeting}})$ factor — Bernoulli rewards the absence of "meeting" (a hammy word). Multinomial ignored it entirely. → SPAM (even more confidently).

Common mistake Steel-manning the common errors

Mistake 1: "Bernoulli should use word counts too." Why it feels right: more occurrences = stronger signal, so surely counting helps. The fix: Bernoulli's whole model is a coin flip per word — its likelihood $p^{x}(1-p)^{1-x}$ is only defined for $x\in\{0,1\}$ . Feeding it counts breaks the probability model. Use Multinomial if counts matter.

Mistake 2: "Absent words don't affect Multinomial, so they never matter." Why it feels right: $p^0=1$ drops them — true. The fix: they do matter in Bernoulli via the $(1-p_t)$ term. Choosing the model changes whether absence is informative.

Mistake 3: forgetting to smooth → zero probabilities. Why it feels right: "The word genuinely never occurred, so 0 is honest." The fix: one zero kills the entire product ( $\log 0=-\infty$ ). Laplace smoothing keeps the estimate finite and generalizes to unseen words.

Mistake 4: denominator confusion. Multinomial divides by $N_c+\alpha|V|$ ( $|V|$ outcomes ); Bernoulli by $D_c+2\alpha$ ( only 2 outcomes ). Mixing them up gives wrong probabilities.

Recall Feynman: explain to a 12-year-old

Imagine sorting letters into "junk" and "real" piles.

Multinomial is like counting how many times each word shows up. The word "FREE!!!" appearing 5 times screams junk louder than appearing once.
Bernoulli just ticks a checklist: "Does the word 'free' show up? Yes/No." It doesn't care if it's 1 or 5 times — and it also notices when an expected friendly word like "meeting" is missing, which is a clue too. Both then say: "Which pile makes this exact set of words most likely?" and pick that pile.

Flashcards

What is the naive assumption in Naive Bayes?

Given the class, all features are conditionally independent, so

P(\mathbf{x}\mid c)=\prod_i P(x_i\mid c)

In Multinomial NB, what does each feature

x_t

represent?

The count (frequency) of word

t

in the document.

In Bernoulli NB, what does each feature

x_t

represent?

A binary indicator: whether word

t

is present (1) or absent (0).

Write the Multinomial NB parameter estimate with Laplace smoothing.

p_{t\mid c}=\dfrac{N_{tc}+\alpha}{N_c+\alpha|V|}

, where

N_{tc}

is total count of word

t

in class

c

Write the Bernoulli NB parameter estimate.

p_{t\mid c}=\dfrac{D_{tc}+\alpha}{D_c+2\alpha}

, where

D_{tc}

is number of docs of class

c

containing word

t

Why is the denominator

D_c+2\alpha

in Bernoulli but

N_c+\alpha|V|

in Multinomial?

Bernoulli word has 2 outcomes (present/absent) →

2\alpha

; Multinomial has

|V|

possible faces →

\alpha|V|

Why does Multinomial ignore absent words but Bernoulli doesn't?

In Multinomial,

p^{x_t}

with

x_t=0

gives 1 (no contribution). In Bernoulli, absent words contribute the factor

(1-p_{t\mid c})

, explicitly penalizing/rewarding absence.

Why do we take logarithms of the Naive Bayes score?

Products of many small probabilities underflow numerically; logs turn products into stable sums.

Why is smoothing (

\alpha>0

) necessary?

A single zero probability makes the whole product 0 (

\log 0=-\infty

), so unseen words would veto a class. Smoothing keeps estimates finite.

Derive the MLE

p_{t\mid c}=N_{tc}/N_c

— what optimization gives it?

Maximize

\sum_t N_{tc}\log p_t

subject to

\sum_t p_t=1

(Lagrange); yields

p_t=N_{tc}/N_c

When should you prefer Multinomial over Bernoulli?

When word frequency carries signal and documents vary in length (typical for longer text / TF features); Bernoulli suits short texts where mere presence matters.

Connections

Bayes Theorem — the underlying inference rule both models apply.
Naive Bayes Classifier — the general framework; these are two likelihood choices.
Gaussian Naive Bayes — the continuous-feature cousin ( $P(x\mid c)$ is a normal density).
Laplace Smoothing — the regularization used in both estimates.
Bag of Words / TF-IDF — feature representations feeding these models.
Log-Sum-Exp trick — numerically stable normalization of the log-scores.
Maximum Likelihood Estimation — source of the parameter formulas.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, dono Naive Bayes text classify karne ke liye same Bayes theorem use karte hain — bas ye difference hai ki word ko kaise dekhte hain. Multinomial NB word ki ginti (count/frequency) dekhta hai: agar "free" 5 baar aaya to signal zyada strong. Isse socho ek loaded dice ki tarah jiske har face pe ek word hai, aur document banane ke liye tum n baar roll karte ho. Isliye formula mein $p_{t|c}=(N_{tc}+1)/(N_c+|V|)$ aata hai — total count divided by total words, plus Laplace smoothing.

Bernoulli NB sirf presence dekhta hai: word aaya ya nahi (1 ya 0). Har word ke liye ek coin flip. Ye ek chhoti par important baat karta hai — jab koi word absent hai to bhi wo score ko affect karta hai, kyunki formula mein $(1-p_t)$ factor lagta hai. Matlab agar "meeting" (ham-type word) missing hai to spam ka score badh jata hai. Multinomial isko ignore kar deta ( $p^0=1$ ).

Smoothing kyun? Agar koi word training mein kabhi nahi dikha, uski probability 0 ho jaati, aur ek bhi 0 poore product ko 0 bana deta ( $\log 0 = -\infty$ ). Isliye $\alpha$ (usually 1) add karte hain — jaise har word ko thoda extra baar dekh liya ho. Multinomial mein denominator mein $\alpha|V|$ add hota (kyunki $|V|$ faces), Bernoulli mein $2\alpha$ (kyunki sirf 2 outcomes: present/absent). Ye mat mix karna, warna probabilities galat ho jaati.

Rule of thumb: lambe documents / frequency matter kare to Multinomial, chhote text jahan sirf word ka hona-na-hona matter kare to Bernoulli. Yaad rakho — "Multi = Many (counts), Bern = Binary".

WHY do we need these at all?

Multinomial Naive Bayes

Deriving the likelihood from first principles

Bernoulli Naive Bayes

Deriving the likelihood

Worked Example (both models on the same data)

Multinomial ( $\alpha=1$ )

Bernoulli ( $\alpha=1$ )

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Test yourself — SVM, Naive Bayes & Probabilistic Models

Connections