Naive Bayes for text classification — Hinglish notes

2.4.11 · AI-ML › SVM, Naive Bayes & Probabilistic Models

WHY hamen yeh chahiye?

WHAT hai problem? Ek document words ka ek bag hai $x = (w_{1}, w_{2}, \dots, w_{n})$ . Hamen woh class $c$ chahiye jo $P (c ∣ x)$ ko maximize kare.

WHY directly $P (c ∣ x)$ model nahi karte? Kyunki $x$ ek bahut bade space mein rehta hai: $V$ size ke vocabulary ke saath, $2^{V}$ possible documents hote hain. Hum har ek ke liye alag probability kabhi estimate nahi kar sakte. Hamen structure chahiye — aur Bayes + independence yeh sasta deta hai.

Classifier ko first principles se derive karna

Step 1 — Hamen sirf classes compare karni hain. Kyunki $P (x)$ $c$ par depend nahi karta, $\overset{c}{^} = ar g max_{c} P (c ∣ x) = ar g max_{c} P (x ∣ c) P (c) .$ Yeh step kyun? Har candidate ko same constant $P (x)$ se divide karne par kabhi nahi badalta ki kaun sabse bada hai, isliye hum isse drop kar dete hain.

Step 2 — Likelihood expand karo. Chain rule se, $P (w_{1}, \dots, w_{n} ∣ c) = P (w_{1} ∣ c) P (w_{2} ∣ c, w_{1}) \dots P (w_{n} ∣ c, w_{1}, \dots, w_{n - 1}) .$ Yeh exact hai lekin parameters ka wahi explosion hai. Hum naive assumption se simplify karte hain.

Step 3 — Naive (conditional independence) assumption. Class diye jane par, har word baaki se independent maana jaata hai: $P (w_{i} ∣ c, w_{1}, \dots, w_{i - 1}) \approx P (w_{i} ∣ c) .$ Yeh step kyun? Yeh $2^{V}$ parameters per class ko sirf $V$ per class tak reduce kar deta hai — hamen sirf ek number chahiye har (word, class) pair ke liye. Yeh "naive" hai kyunki word order/co-occurrence ignore hota hai.

Parameters estimate karna (Multinomial model)

HOW hamen $P (w ∣ c)$ milta hai? Class $c$ ke training documents mein words count karke.

Zero-probability disaster. Agar ek word training mein class $c$ ke saath kabhi nahi aaya, $P (w ∣ c) = 0$ , aur poora product $0$ ban jaata hai — ek unseen word poori class ko veto kar deta hai.

Figure — Naive Bayes for text classification

Worked example 1 — Spam filter

Training (word tokens):

Class	docs	words
spam	2	"free money now", "free free win"
ham	2	"meeting now please", "please review"

Vocabulary $V = {$ free, money, now, win, meeting, please, review $}$ , $∣ V ∣ = 7$ .

Counts: spam mein 6 tokens hain ( $N_{s p am} = 6$ ): free×3, money×1, now×1, win×1.
ham mein 5 tokens hain ( $N_{ham} = 5$ ): meeting×1, now×1, please×2, review×1.
Priors: $P (s p am) = P (ham) = 0.5$ .

"free now" classify karo. Laplace $α = 1$ use karo, denominator spam $= 6 + 7 = 13$ , ham $= 5 + 7 = 12$ .

$P (free ∣ s p am) = \frac{3 + 1}{13} = \frac{4}{13}, P (now ∣ s p am) = \frac{1 + 1}{13} = \frac{2}{13}$ $P (free ∣ ham) = \frac{0 + 1}{12} = \frac{1}{12}, P (now ∣ ham) = \frac{1 + 1}{12} = \frac{2}{12}$

Scores (equal prior 0.5 drop karo): $s p am : \frac{4}{13} \cdot \frac{2}{13} = \frac{8}{169} = 0.0473, ham : \frac{1}{12} \cdot \frac{2}{12} = \frac{2}{144} = 0.0139.$ Yeh step kyun? Hum $\prod P (w ∣ c) P (c)$ compare karte hain; spam jeet ta hai → spam classify kiya gaya. ✓ (word "free" ne drive kiya, "now" neutral tha).

Worked example 2 — Log form kyun safe hai

Ek 100-word document jisme har $P (w ∣ c) \approx 1 0^{- 3}$ hai, product $\approx 1 0^{- 300}$ deta hai, jo floating point mein $0.0$ underflow kar jaata hai. Logs use karke: $\sum lo g (1 0^{- 3}) = 100 \times (- 6.9) = - 690$ , ek perfectly representable number. Kyun? Logs khatarnak chhote products ko safe sums mein convert kar dete hain.

Bernoulli vs Multinomial (antar jaano)

Common mistakes (Steel-manned)

Active recall

Recall Khud test karo (answers hide karo)

$P (x)$ kyun drop kar sakte hain? → Yeh sabhi classes mein constant hai; argmax affect nahi karta.
Naive assumption kya hai? → Words class diye jane par conditionally independent hain.
Logs kyun lete hain? → Underflow rokne ke liye; tiny products ki jagah safe sums; argmax unchanged.
Smoothing ke bina kya toot ta hai? → Unseen word se zero probability poore product ko 0 kar deti hai.
Multinomial vs Bernoulli feature? → Counts vs binary presence.

Recall Feynman: ek 12-saal ke bacche ko samjhao

Socho do toy boxes hain. Box A ("junk mail") free, win, money jaise words se bhari hai. Box B ("friends") mein meeting, please, review hain. Koi tumhe ek note deta hai. Tum har word dekhte ho aur guess karte ho ki woh probably kis box se aaya, aur ek running tally rakhte ho. Jo box zyada words "fit" karta hai woh jeetta hai. Yeh "naive" hai kyunki hum maan lete hain ki har word alag se choose hota hai, yeh ignore karke ki kuch words saath chalte hain — lekin phir bhi yeh surprisingly aksar sahi box guess kar leta hai.

The 80/20 core (jo actually matter karta hai)

$\overset{c}{^} = ar g max_{c} P (c) \prod P (w_{i} ∣ c)$ — poora method.
Laplace smoothing taaki koi probability zero na ho.
Log-space mein kaam karo.

Yeh teen master karo aur tum NB text classification implement kar sakte ho.

Naive Bayes classification rule

\overset{c}{^} = ar g max_{c} P (c) \prod_{i} P (w_{i} ∣ c)

Bayes' rule mein classification ke liye

P (x)

kyun ignore kar sakte hain

Yeh har class ke liye same hota hai, isliye argmax kabhi nahi badalta.

"Naive" assumption kya hai

Words class diye jane par conditionally independent hain:

P (w_{i} ∣ c, others) = P (w_{i} ∣ c)

NB false independence assumption ke bawajood kyun kaam karta hai

Isko sirf chahiye ki sahi class ka score sabse bada ho; ranking biased probability estimates ke bawajood survive karti hai.

Laplace smoothing formula

P (w ∣ c) = \frac{N _{w c} + α}{N _{c} + α ∣ V ∣}

Denominator mein

α ∣ V ∣

kyun add karte hain

Humne

∣ V ∣

vocabulary words mein se har ek mein

α

add kiya, isliye kul added mass

α ∣ V ∣

hai, jo distribution ko normalized rakhta hai.

Probabilities ke logarithms kyun lete hain

Kai tiny probabilities ka product 0 tak underflow kar jaata hai; logs unhe safe sum mein convert kar dete hain, aur log monotonic hai isliye argmax unchanged rehta hai.

Smoothing ke bina class score kya zero kar deta hai

Ek bhi word jo us class ke saath kabhi nahi dekha gaya,

P (w ∣ c) = 0

deta hai, jo poora product 0 bana deta hai.

Multinomial vs Bernoulli NB feature type

Multinomial word counts use karta hai; Bernoulli binary presence/absence use karta hai (aur absent words model karta hai).

Class c ke liye prior estimate

P (c) = \frac{# docs in c}{# total docs}

P (w ∣ c)

ke liye MLE (unsmoothed)

\frac{N _{w c}}{N _{c}}

= class mein word count divided by class mein kul tokens.

Connections

Bayes Theorem — woh engine jo likelihood ko posterior mein flip karta hai.
Maximum Likelihood Estimation — counts probabilities kaise bante hain.
Laplace Smoothing — unseen events / additive priors handle karna.
Bag of Words Model — document representation jo NB assume karta hai.
TF-IDF — alternative weighting jo raw counts se aksar compare hoti hai.
Logistic Regression — generative NB ka discriminative counterpart.
SVM for text classification — same chapter mein margin-based alternative.
Underflow and Log-space computation — numerical-stability trick jo har jagah reuse hoti hai.