Laplace smoothing

2.4.10 · AI-ML › SVM, Naive Bayes & Probabilistic Models

WHY chahiye yeh humein?

Naive Bayes classify karta hai is tarah:

P (C ∣ x_{1}, \dots, x_{n}) \propto P (C) i = 1 \prod n P (x_{i} ∣ C)

Ek conditional ka Maximum Likelihood estimate sirf ek count ratio hota hai:

P_{M L} (x_{i} = v ∣ C) = \frac{count ( x _{i} = v , C )}{count ( C )}

HOW derive karen isse (Bayesian first principles)

$K$ outcomes ke counts ko Multinomial se model karo jisme unknown probabilities $θ = (θ_{1}, \dots, θ_{K})$ hain. Ek Dirichlet prior rakho jisme saare parameters $α$ ke barabar hain:

θ \sim Dir (α, \dots, α), p (θ) \propto v \prod θ_{v}^{α - 1}

Counts $c_{1}, \dots, c_{K}$ observe karo (total $N$ ). Dirichlet Multinomial ka conjugate hai, isliye posterior bhi Dirichlet hai:

θ ∣ data \sim Dir (c_{1} + α, \dots, c_{K} + α)

$θ_{v}$ ka posterior mean (natural point estimate) hai:

E [θ_{v} ∣ data] = \frac{c _{v} + α}{\sum _{u} ( c _{u} + α )} = \frac{c _{v} + α}{N + α K}

Worked Example 1 — Spam word

Vocabulary size $K = 5$ : {buy, free, meeting, project, lunch}. Spam class mein humne total $N = 8$ words dekhe, counts hain: buy=4, free=3, meeting=1, project=0, lunch=0.

"project" ke liye raw MLE:

P_{M L} (project ∣ spam) = \frac{0}{8} = 0 \to product ko khatam kar deta hai

Yeh step kyun? Failure dikhata hai: "project" wala koi bhi email spam-score 0 paayega, doosre words se regardless.

Add-one ( $α = 1$ ):

P (project ∣ spam) = \frac{0 + 1}{8 + 1 \cdot 5} = \frac{1}{13} \approx 0.077

Yeh step kyun? Denominator hai $N + α K = 8 + 5$ . Ab impossible sirf unlikely ho jaata hai.

"buy" check karo:

P (buy ∣ spam) = \frac{4 + 1}{8 + 5} = \frac{5}{13} \approx 0.385

Yeh step kyun? Smoothing confident estimates ko thoda uniform ( $1/5 = 0.2$ ) ki taraf shrink karta hai, jo desirable regularization hai.

Sum check: $\frac{5 + 4 + 2 + 1 + 1}{13} = \frac{13}{13} = 1$ ✓ Yeh step kyun? Confirm karta hai ki yeh valid distribution hai — $α K$ use karne ka yahi point hai.

Worked Example 2 — $α$ ka effect

Wohi "project" count = 0, $N = 8$ , $K = 5$ :

$α$	$P (project ∣ spam) = \frac{0 + α}{8 + 5 α}$
$0.001$	$\approx 0.000125$ (barely smoothed)
$1$	$1/13 \approx 0.077$
$100$	$100/508 \approx 0.197 \approx 1/5$ (almost uniform)

Yeh step kyun? Jaise $α \to \infty$ , har estimate $\to \frac{1}{K}$ (max smoothing, data ignore). Jaise $α \to 0$ , raw MLE milta hai. Toh $α$ ek bias–variance knob hai, jo cross-validation se tune hota hai.

Worked Example 3 — Test word jo kabhi dekha hi nahi

Maan lo ek test email mein "hello" hai, jo kisi bhi class ki vocabulary mein nahi hai. Handle karne ka tarika: ya toh unknown tokens drop karo, ya ek <UNK> slot reserve karo taaki $K$ usse count kare. Agar count 0 ke saath include kiya:

P (hello ∣ spam) = \frac{0 + 1}{N _{s p am} + ( K + 1 )}

Yeh step kyun? Consistency: tumhe wahi $K$ (UNK slot sameto) dono classes ke numerator normalization aur har class mein use karna chahiye, warna alag classes ke probabilities alag scales par hoti hain.

Recall Feynman: ek 12-saal ke bachche ko samjhao

Socho tum guess kar rahe ho ki kaunse cereal box mein toy aata hai. Tumne 8 boxes khole, kuch toys dekhe lekin kabhi robot nahi dekha. Kya iska matlab robot impossible hai? Nahi! Bas tumne itne boxes nahi khole. Toh shuru mein, tum pretend karo ki tumhe pehle se har ek toy ek baar mil chuki hai. Ab kuch bhi "impossible" nahi, bas rare hai. Yahi Laplace smoothing hai — har cheez ki ek imaginary sighting add karo taaki koi surprise tumhara poora guess na todo.

Forecast-then-Verify

Aage padhne se pehle predict karo: agar main vocabulary $K$ double kar dun lekin counts same rakhun, toh zero-count word ki smoothed probability upar jaayegi ya neeche? → Neeche, kyunki denominator $N + α K$ badh jaata hai jabki numerator $α$ sama rehta hai — zyada possible outcomes matlab har unseen ek individually zyada rare. (VERIFY block mein verified.)

Flashcards

Unsmoothed Naive Bayes unseen feature values par kyun fail karta hai?

Conditional probability 0 ho jaati hai, aur kyunki NB conditionals multiply karta hai, ek 0 poori class posterior ko 0 bana deta hai.

Laplace-smoothed conditional probability likho.

P (x_{i} = v ∣ C) = \frac{count ( x _{i} = v , C ) + α}{count ( C ) + α K}

Denominator mein

α K

(na ki

α

) kyun add karte hain?

Smoothed distribution ko normalized rakhne ke liye —

K

outcomes par sum exactly 1 aata hai.

Laplace smoothing mein

K

kya hai?

Feature ki possible values ki sankhya (jaise text ke liye vocabulary size).

Add-one smoothing kya hai?

α = 1

ke saath Laplace smoothing.

Lidstone smoothing kya hai?

General

0 < α < 1

ke saath Laplace smoothing.

Kaun sa Bayesian model exactly Laplace smoothing deta hai?

Multinomial likelihood with symmetric Dirichlet(

α, \dots, α

) prior; smoothed estimate posterior mean hai.

Jaise

α \to \infty

kya hota hai?

Har estimate uniform

1/ K

ki taraf jaata hai (maximum smoothing, data ignore ho jaata hai).

Jaise

α \to 0

kya hota hai?

Raw Maximum Likelihood count ratio milta hai (koi smoothing nahi).

α

ko tune kaise karte hain?

Cross-validation se; yeh ek bias–variance regularization knob hai.

Connections

Naive Bayes Classifier — smoothing iske conditional probability estimates ke andar rehti hai.
Maximum Likelihood Estimation — unsmoothed baseline ( $α = 0$ ).
Dirichlet Distribution & Conjugate Priors — add- $α$ ka Bayesian origin.
Bias-Variance Tradeoff — $α$ isko control karta hai.
Bag of Words / Text Classification — jahan $K$ = vocabulary size.
Log probabilities — numerical underflow se bachne ke liye smoothing ke saath use hota hai.