Naive Bayes for text classification — AI-ML notes

WHY do we need this?

WHAT is the problem? A document is a bag of words $\mathbf{x}=(w_1,w_2,\dots,w_n)$ . We want the class $c$ that maximizes $P(c\mid \mathbf{x})$ .

WHY not model $P(c\mid\mathbf{x})$ directly? Because $\mathbf{x}$ lives in a gigantic space: with a vocabulary of size $V$ , there are $2^V$ possible documents. We could never estimate a separate probability for each. We need structure — and Bayes + independence gives it cheaply.

Deriving the classifier from first principles

Step 1 — We only need to compare classes. Since $P(\mathbf{x})$ does not depend on $c$ , $\hat c = \arg\max_c P(c\mid\mathbf{x}) = \arg\max_c P(\mathbf{x}\mid c)\,P(c).$ Why this step? Dividing every candidate by the same constant $P(\mathbf{x})$ never changes which one is largest, so we drop it.

Step 2 — Expand the likelihood. By the chain rule, $P(w_1,\dots,w_n\mid c)=P(w_1\mid c)\,P(w_2\mid c,w_1)\cdots P(w_n\mid c,w_1,\dots,w_{n-1}).$ This is exact but has the same explosion of parameters. We simplify with the naive assumption.

Step 3 — The naive (conditional independence) assumption. Given the class, each word is assumed independent of the others: $P(w_i\mid c, w_1,\dots,w_{i-1}) \approx P(w_i\mid c).$ Why this step? It reduces $2^V$ parameters per class to just $V$ per class — we only need one number per (word, class) pair. It's "naive" because word order/co-occurrence is ignored.

Estimating the parameters (Multinomial model)

HOW do we get $P(w\mid c)$ ? By counting words in the training documents of class $c$ .

The zero-probability disaster. If a word never appeared with class $c$ in training, $P(w\mid c)=0$ , and the whole product becomes $0$ — one unseen word vetoes the entire class.

Figure — Naive Bayes for text classification

Worked example 1 — Spam filter

Training (word tokens):

Class	docs	words
spam	2	"free money now", "free free win"
ham	2	"meeting now please", "please review"

Vocabulary $V=\{$ free, money, now, win, meeting, please, review $\}$ , $|V|=7$ .

Counts: spam has 6 tokens ( $N_{spam}=6$ ): free×3, money×1, now×1, win×1.
ham has 5 tokens ( $N_{ham}=5$ ): meeting×1, now×1, please×2, review×1.
Priors: $P(spam)=P(ham)=0.5$ .

Classify "free now". Use Laplace $\alpha=1$ , denominator spam $=6+7=13$ , ham $=5+7=12$ .

$P(\text{free}\mid spam)=\frac{3+1}{13}=\frac{4}{13},\quad P(\text{now}\mid spam)=\frac{1+1}{13}=\frac{2}{13}$ $P(\text{free}\mid ham)=\frac{0+1}{12}=\frac{1}{12},\quad P(\text{now}\mid ham)=\frac{1+1}{12}=\frac{2}{12}$

Scores (drop equal prior 0.5): $spam:\ \tfrac{4}{13}\cdot\tfrac{2}{13}=\tfrac{8}{169}=0.0473,\qquad ham:\ \tfrac{1}{12}\cdot\tfrac{2}{12}=\tfrac{2}{144}=0.0139.$ Why this step? We compare $\prod P(w\mid c)P(c)$ ; spam wins → classified as spam. ✓ (the word "free" drove it, "now" was neutral).

Worked example 2 — Why the log form is safer

A 100-word document with each $P(w\mid c)\approx 10^{-3}$ gives a product $\approx10^{-300}$ , which underflows to $0.0$ in floating point. Using logs: $\sum \log(10^{-3}) = 100\times(-6.9)=-690$ , a perfectly representable number. Why? Logs turn dangerous tiny products into safe sums.

Bernoulli vs Multinomial (know the difference)

Common mistakes (Steel-manned)

Active recall

Recall Test yourself (hide answers)

Why can we drop $P(\mathbf{x})$ ? → It's constant across classes; doesn't affect argmax.
What is the naive assumption? → Words conditionally independent given the class.
Why take logs? → Prevent underflow; sums instead of tiny products; argmax unchanged.
What breaks without smoothing? → Zero probability from unseen word zeroes the product.
Multinomial vs Bernoulli feature? → Counts vs binary presence.

Recall Feynman: explain to a 12-year-old

Imagine two toy boxes. Box A ("junk mail") is full of words like free, win, money. Box B ("friends") has meeting, please, review. Someone hands you a note. You look at each word and guess which box it probably came from, and you keep a running tally. The box that "fits" more of the words wins. It's "naive" because we pretend each word is picked separately, ignoring that some words like to travel together — but it still guesses the right box surprisingly often.

The 80/20 core (what actually matters)

$\hat c=\arg\max_c P(c)\prod P(w_i\mid c)$ — the whole method.
Laplace smoothing so no probability is zero.
Work in log-space.

Master those three and you can implement NB text classification.

Naive Bayes classification rule

\hat c=\arg\max_c P(c)\prod_i P(w_i\mid c)

Why can we ignore

P(\mathbf{x})

in Bayes' rule for classification

It is the same for every class, so it never changes the argmax.

What is the "naive" assumption

Words are conditionally independent given the class:

P(w_i\mid c,\text{others})=P(w_i\mid c)

Why does NB work despite the false independence assumption

It only needs the correct class to have the largest score; the ranking survives biased probability estimates.

Laplace smoothing formula

P(w\mid c)=\dfrac{N_{wc}+\alpha}{N_c+\alpha|V|}

Why add

\alpha|V|

in the denominator

We added

\alpha

to each of the

|V|

vocabulary words, so total added mass is

\alpha|V|

, keeping the distribution normalized.

Why take logarithms of the probabilities

Products of many tiny probabilities underflow to 0; logs turn them into a safe sum, and log is monotonic so argmax is unchanged.

What zeroes out a class score without smoothing

A single word never seen with that class gives

P(w\mid c)=0

, making the whole product 0.

Multinomial vs Bernoulli NB feature type

Multinomial uses word counts; Bernoulli uses binary presence/absence (and models absent words).

Prior estimate for class c

P(c)=\dfrac{\#\text{docs in }c}{\#\text{total docs}}

MLE for

P(w\mid c)

(unsmoothed)

\dfrac{N_{wc}}{N_c}

= word count in class over total tokens in class.

Connections

Bayes Theorem — the engine that flips likelihood into posterior.
Maximum Likelihood Estimation — how counts become probabilities.
Laplace Smoothing — handling unseen events / additive priors.
Bag of Words Model — the document representation NB assumes.
TF-IDF — alternative weighting often compared with raw counts.
Logistic Regression — the discriminative counterpart to generative NB.
SVM for text classification — margin-based alternative in the same chapter.
Underflow and Log-space computation — numerical-stability trick reused everywhere.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Naive Bayes ka core idea simple hai: hume decide karna hai ki koi document (email, message) kis class ka hai — jaise spam ya ham. Hum Bayes theorem use karte hain jo "words given class" ko "class given words" me flip kar deta hai. Formula banta hai: har class ke liye us class ka prior le lo, aur us document ke saare words ki probabilities multiply kar do. Jis class ka score sabse bada, wahi answer.

"Naive" isliye bola jaata hai kyunki hum assume karte hain ki words ek dusre se independent hain given the class — matlab "New" aur "York" saath aate hain ye baat ignore kar dete hain. Ye assumption technically galat hai, par magic ye hai ki classification me sirf argmax matter karta hai, exact probability nahi. Isliye ranking sahi aa jaati hai aur accuracy achhi rehti hai.

Do practical cheezein yaad rakho. Pehli: Laplace smoothing — agar koi word training me kisi class ke saath kabhi nahi aaya, to uski probability 0 ho jaayegi aur poora product 0 ho jaayega, ek word puri class ko maar dega. Isliye $+\alpha$ upar aur $+\alpha|V|$ neeche daalte hain. Dusri: log lena — bahut saari choti probabilities multiply karne se number itna chota ho jaata hai ki computer usko 0 samajh leta hai (underflow). Log lene se multiply, addition ban jaata hai, aur log monotonic hone ki wajah se argmax same rehta hai.

Yaad rakhne ka tarika: Prior × Likelihood, phir Log, phir Laplace. Bas yahi teen cheez pakad lo aur tum spam filter khud bana sakte ho — yahi 80/20 hai.

Test yourself — SVM, Naive Bayes & Probabilistic Models

Connections