Laplace smoothing

WHY do we need it?

Naive Bayes classifies using:

P(C \mid x_1,\dots,x_n) \propto P(C)\prod_{i=1}^{n} P(x_i \mid C)

The Maximum Likelihood estimate for one conditional is just a count ratio:

P_{ML}(x_i = v \mid C) = \frac{\text{count}(x_i=v, C)}{\text{count}(C)}

HOW to derive it (Bayesian first principles)

Model the counts of the $K$ outcomes as a Multinomial with unknown probabilities $\theta=(\theta_1,\dots,\theta_K)$ . Put a Dirichlet prior with all parameters equal to $\alpha$ :

\theta \sim \text{Dir}(\alpha,\dots,\alpha),\qquad p(\theta)\propto \prod_{v}\theta_v^{\alpha-1}

Observe counts $c_1,\dots,c_K$ (total $N$ ). The Dirichlet is conjugate to the Multinomial, so the posterior is again Dirichlet:

\theta \mid \text{data} \sim \text{Dir}(c_1+\alpha,\dots,c_K+\alpha)

The posterior mean of $\theta_v$ (the natural point estimate) is:

\mathbb{E}[\theta_v \mid \text{data}] = \frac{c_v+\alpha}{\sum_{u}(c_u+\alpha)} = \frac{c_v+\alpha}{N+\alpha K}

Worked Example 1 — Spam word

Vocabulary size $K=5$ : {buy, free, meeting, project, lunch}. In the spam class we saw $N=8$ words total, with counts: buy=4, free=3, meeting=1, project=0, lunch=0.

Raw MLE for "project":

P_{ML}(\text{project}\mid \text{spam}) = \frac{0}{8}=0 \quad \text{→ kills the product}

Why this step? Shows the failure: any email containing "project" would get spam-score 0 regardless of other words.

Add-one ( $\alpha=1$ ):

P(\text{project}\mid \text{spam}) = \frac{0+1}{8+1\cdot 5}=\frac{1}{13}\approx 0.077

Why this step? Denominator is $N+\alpha K = 8+5$ . Now the impossible becomes merely unlikely.

Check "buy":

P(\text{buy}\mid\text{spam})=\frac{4+1}{8+5}=\frac{5}{13}\approx 0.385

Why this step? Smoothing shrinks confident estimates slightly toward uniform ( $1/5=0.2$ ), which is desirable regularization.

Sum check: $\frac{5+4+2+1+1}{13}=\frac{13}{13}=1$ ✓ Why this step? Confirms it's a valid distribution — the whole point of using $\alpha K$ .

Worked Example 2 — Effect of $\alpha$

Same "project" count = 0, $N=8$ , $K=5$ :

$\alpha$	$P(\text{project}\mid\text{spam}) = \dfrac{0+\alpha}{8+5\alpha}$
$0.001$	$\approx 0.000125$ (barely smoothed)
$1$	$1/13 \approx 0.077$
$100$	$100/508 \approx 0.197 \approx 1/5$ (almost uniform)

Why this step? As $\alpha\to\infty$ , every estimate $\to \frac1K$ (max smoothing, ignores data). As $\alpha\to 0$ , we recover raw MLE. So $\alpha$ is a bias–variance knob, tuned by cross-validation.

Worked Example 3 — Test word never seen at all

Suppose a test email contains "hello", which is not in the vocabulary for either class. Handling: either drop unknown tokens, or reserve an <UNK> slot so $K$ counts it. If included with count 0:

P(\text{hello}\mid\text{spam})=\frac{0+1}{N_{spam}+ (K{+}1)}

Why this step? Consistency: you must use the same $K$ (including the UNK slot) in both numerator normalization and every class, otherwise probabilities across classes are on different scales.

Recall Feynman: explain to a 12-year-old

Imagine you're guessing which cereal box a toy comes in. You opened 8 boxes, saw some toys but never saw a robot. Does that mean a robot is impossible? No! You just haven't opened enough boxes. So before you start, you pretend you already found one of every toy. Now nothing is "impossible," just rare. That's Laplace smoothing — adding one imaginary sighting of everything so a surprise doesn't break your whole guess.

Forecast-then-Verify

Before reading further, predict: if I double the vocabulary $K$ but keep counts fixed, does a zero-count word's smoothed probability go up or down? → Down, because denominator $N+\alpha K$ grows while numerator $\alpha$ stays — more possible outcomes means each unseen one is individually rarer. (Verified in the VERIFY block.)

Flashcards

Why does unsmoothed Naive Bayes fail on unseen feature values?

The conditional probability becomes 0, and since NB multiplies conditionals, one 0 makes the entire class posterior 0.

Write the Laplace-smoothed conditional probability.

P(x_i=v\mid C)=\dfrac{\text{count}(x_i=v,C)+\alpha}{\text{count}(C)+\alpha K}

Why is

\alpha K

(not

\alpha

) added to the denominator?

To keep the smoothed distribution normalized — summing over the

K

outcomes gives exactly 1.

What is

K

in Laplace smoothing?

The number of possible values the feature can take (e.g. vocabulary size for text).

What is add-one smoothing?

Laplace smoothing with

\alpha=1

What is Lidstone smoothing?

Laplace smoothing with a general

0<\alpha<1

What Bayesian model yields Laplace smoothing exactly?

Multinomial likelihood with a symmetric Dirichlet(

\alpha,\dots,\alpha

) prior; the smoothed estimate is the posterior mean.

What happens as

\alpha\to\infty

Every estimate approaches the uniform

1/K

(maximum smoothing, data ignored).

What happens as

\alpha\to 0

You recover the raw Maximum Likelihood count ratio (no smoothing).

How do you tune

\alpha

Via cross-validation; it's a bias–variance regularization knob.

Connections

Naive Bayes Classifier — smoothing lives inside its conditional probability estimates.
Maximum Likelihood Estimation — the unsmoothed baseline ( $\alpha=0$ ).
Dirichlet Distribution & Conjugate Priors — the Bayesian origin of add- $\alpha$ .
Bias-Variance Tradeoff — $\alpha$ controls it.
Bag of Words / Text Classification — where $K$ = vocabulary size.
Log probabilities — used alongside smoothing to avoid numerical underflow.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, Naive Bayes ka simple funda hai: har class ke liye hum saare feature probabilities ko multiply karte hain. Ab problem yeh hai — agar training data mein koi word kisi class ke saath ek baar bhi nahi dikha, to uski probability seedha 0 ban jaati hai. Aur 0 ko kisi bhi cheez se multiply karo, poora answer 0 ho jaata hai. Matlab ek anjaan word puri prediction ko barbaad kar deta hai. Yeh overconfident behaviour hai — "maine nahi dekha, iska matlab impossible hai" — jo galat hai.

Laplace smoothing isko fix karta hai bahut hi cute tareeke se: har outcome ko pehle se hi thoda count de do, jaise humne har cheez ek baar (ya $\alpha$ baar) dekh li ho. Formula banta hai $\frac{count+\alpha}{N+\alpha K}$ . Yahan $K$ = total possible values (text mein vocabulary size). Neeche $\alpha K$ isliye add karte hain taaki saari probabilities milke exactly 1 banein — warna distribution valid hi nahi rahega.

Deep baat yeh hai ki yeh koi jugaad nahi hai — yeh actually ek Dirichlet prior ke saath Bayesian estimate ka posterior mean hai. $\alpha$ ek knob hai: chhota $\alpha$ = data pe zyada bharosa (kam smoothing), bada $\alpha$ = sab kuch uniform ( $1/K$ ) ki taraf khinch jaata hai (zyada smoothing). Isko cross-validation se tune karte ho. Bas yaad rakhna: "+ $\alpha$ upar, + $\alpha K$ neeche" — zero kabhi nahi aayega, aur sab kuch sum-to-one rahega. Simple, powerful, exam-favourite.

WHY do we need it?

WHAT is Laplace smoothing?

HOW to derive it (Bayesian first principles)

Worked Example 1 — Spam word

Worked Example 2 — Effect of $\alpha$

Worked Example 3 — Test word never seen at all

Forecast-then-Verify

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Test yourself — SVM, Naive Bayes & Probabilistic Models

Connections