Gaussian Naive Bayes — AI-ML notes

WHY does this method exist?

We want $P(\text{class} \mid \text{features})$ . That is hard to model directly for continuous data. Bayes' theorem lets us flip it into things we can estimate from data:

$P(y \mid \mathbf{x}) = \frac{P(y)\,P(\mathbf{x}\mid y)}{P(\mathbf{x})}$

WHAT we want: the posterior $P(y\mid\mathbf{x})$ .
WHAT we can measure: the prior $P(y)$ (just count classes) and the likelihood $P(\mathbf{x}\mid y)$ (how features are distributed inside each class).

The problem: $P(\mathbf{x}\mid y)=P(x_1,x_2,\dots,x_d\mid y)$ is a joint distribution over all features — hopeless to estimate with limited data. The "Naive" trick fixes this.

The Naive assumption (the 80/20 core)

WHY is this a huge win? A joint distribution over $d$ features needs exponentially many parameters. Splitting it into $d$ separate 1-D distributions needs only $O(d)$ parameters. It is "naive" because features are rarely truly independent — but the classifier still works surprisingly well.

The Gaussian part: modelling each $P(x_j\mid y)$

We only need two numbers per feature per class: the mean $\mu_{y,j}$ and variance $\sigma_{y,j}^2$ .

HOW do we estimate $\mu$ and $\sigma^2$ ? (Derivation from scratch)

We use Maximum Likelihood Estimation (MLE). Take all $n_y$ training points in class $y$ ; their feature- $j$ values are $x^{(1)},\dots,x^{(n_y)}$ . The log-likelihood of a Gaussian is:

$\ell(\mu,\sigma^2)=\sum_{i=1}^{n_y}\left[-\tfrac12\ln(2\pi\sigma^2)-\frac{(x^{(i)}-\mu)^2}{2\sigma^2}\right]$

Solve for $\mu$ — set $\partial\ell/\partial\mu=0$ : $\frac{\partial\ell}{\partial\mu}=\sum_i \frac{x^{(i)}-\mu}{\sigma^2}=0 \;\Rightarrow\; \boxed{\mu = \frac{1}{n_y}\sum_i x^{(i)}}$ Why this step? The variance cancels, leaving the plain sample mean.

Solve for $\sigma^2$ — set $\partial\ell/\partial\sigma^2=0$ : $\frac{\partial\ell}{\partial\sigma^2}=\sum_i\left[-\frac{1}{2\sigma^2}+\frac{(x^{(i)}-\mu)^2}{2\sigma^4}\right]=0 \;\Rightarrow\; \boxed{\sigma^2=\frac{1}{n_y}\sum_i (x^{(i)}-\mu)^2}$ Why this step? Multiply through by $2\sigma^4$ and rearrange — you get the sample variance.

So training is just: compute per-class, per-feature mean and variance. That's it.

The decision rule (put it together)

Since $P(\mathbf{x})$ is the same for every class, we drop it and maximise the numerator:

$\hat{y}=\arg\max_y \; P(y)\prod_{j=1}^{d}P(x_j\mid y)$

Worked Example 1 — one feature, two classes

Feature = height (cm). Class A (short): $\mu_A=160,\ \sigma_A=5$ . Class B (tall): $\mu_B=180,\ \sigma_B=5$ . Priors equal. Classify $x=172$ .

$\ln P(x\mid A)\propto -\frac{(172-160)^2}{2\cdot25}=-\frac{144}{50}=-2.88$ Why? Constant $\ln(2\pi\sigma^2)$ is identical for both classes (same $\sigma$ ), so it cancels.
$\ln P(x\mid B)\propto -\frac{(172-180)^2}{2\cdot25}=-\frac{64}{50}=-1.28$

$-1.28 > -2.88 \Rightarrow$ Class B (tall). Why? $172$ is closer to $180$ ; with equal $\sigma$ and equal priors, GNB just picks the nearest mean.

Worked Example 2 — unequal variance flips intuition

Class A: $\mu_A=160,\sigma_A=20$ . Class B: $\mu_B=180,\sigma_B=3$ . Classify $x=172$ .

$\ln P(x\mid A)=-\tfrac12\ln(2\pi\cdot400)-\frac{144}{800}=-4.03-0.18=-4.21$
$\ln P(x\mid B)=-\tfrac12\ln(2\pi\cdot9)-\frac{64}{18}=-2.02-3.56=-5.58$

$-4.21 > -5.58 \Rightarrow$ Class A, even though $172$ is closer to B's mean! Why this step? B is very "tight" ( $\sigma=3$ ): a value 12 units away is extremely unlikely for B. The $-\tfrac12\ln(2\pi\sigma^2)$ term and the variance in the denominator both penalise confident-but-far predictions. This is why GNB decision boundaries are curved (quadratic), not just midpoints.

Worked Example 3 — two features

Point $\mathbf{x}=(x_1,x_2)$ . GNB scores $\ln P(y)-\frac{(x_1-\mu_{y1})^2}{2\sigma_{y1}^2}-\frac{(x_2-\mu_{y2})^2}{2\sigma_{y2}^2}+\text{const}_y$ . Why? The independence assumption turns the joint into a sum of per-feature log-terms — each feature votes separately.

Common mistake Steel-manning the common errors

(1) "Naive = features are actually independent." Feels right because that's the assumption stated. Fix: it's an assumption we impose, usually false; GNB works despite it because argmax is robust.

(2) "GNB picks the nearest class mean." Feels right and is true only when all variances/priors are equal (Example 1). Fix: unequal $\sigma$ or priors bends the boundary (Example 2). The variance and $\ln\sigma$ terms matter.

(3) "Multiply the raw probabilities." Feels natural. Fix: with many features you underflow to $0.0$ ; always sum logs.

(4) "Use $n-1$ (unbiased) variance." Feels statistically correct. Fix: MLE gives $n$ in the denominator; scikit-learn also adds a tiny var_smoothing to $\sigma^2$ to avoid divide-by-zero when a feature is constant.

Recall Feynman: explain to a 12-year-old

Imagine each class is a person throwing darts at a number line, aiming for their favourite spot (the mean). Some throwers are shaky (big spread) and some are super precise (small spread). A new dart lands. You guess which thrower threw it by asking: "For each thrower, how surprised would I be if THEY threw it here?" You pick the thrower who'd be least surprised — considering both how close it is to their target and how precise they usually are. The precise thrower gets little credit for far-away darts; the shaky one is forgiven. That "least surprised" rule is Gaussian Naive Bayes.

Flashcards

What core assumption makes Naive Bayes "naive"?

Features are conditionally independent given the class, so

P(\mathbf{x}\mid y)=\prod_j P(x_j\mid y)

Why does the naive assumption help computationally?

It reduces a joint distribution (exponential parameters) to

d

separate 1-D distributions (

O(d)

parameters).

In Gaussian NB, what distribution models each

P(x_j\mid y)

A univariate normal with class/feature-specific mean

\mu_{y,j}

and variance

\sigma_{y,j}^2

What are the MLE estimates for

\mu

and

\sigma^2

\mu=\frac1n\sum x_i

(sample mean) and

\sigma^2=\frac1n\sum(x_i-\mu)^2

(sample variance, divide by

n

Why do we drop

P(\mathbf{x})

in the decision rule?

It's the same for all classes, so it doesn't change the argmax.

Why take logs of the probabilities?

Products of many small numbers underflow; log is monotonic so argmax is preserved and products become sums.

When does GNB reduce to "nearest class mean"?

When all classes share equal variance and equal priors.

Why is the GNB decision boundary generally quadratic?

The

(x-\mu)^2/2\sigma^2

and

-\tfrac12\ln\sigma^2

terms differ per class, giving a quadratic (curved) boundary.

What is var_smoothing for?

A small value added to variances to prevent division by zero when a feature has near-zero variance.

Full GNB decision rule (log form)?

\hat y=\arg\max_y[\ln P(y)+\sum_j(-\tfrac12\ln(2\pi\sigma_{y,j}^2)-\frac{(x_j-\mu_{y,j})^2}{2\sigma_{y,j}^2})]

Connections

Bayes Theorem — the foundation GNB rewrites.
Naive Bayes — the general family (Gaussian for continuous, Multinomial Naive Bayes for counts, Bernoulli Naive Bayes for binary).
Maximum Likelihood Estimation — how $\mu,\sigma^2$ are derived.
Gaussian Distribution — the likelihood model.
LDA & QDA — GNB is essentially QDA with a diagonal covariance (independent features).
Logistic Regression — a discriminative cousin that models $P(y\mid\mathbf{x})$ directly.
Log-Sum-Exp trick — stable posterior normalisation.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Gaussian Naive Bayes ka core idea simple hai: har class ko ek bell-curve (normal distribution) maan lo. Training ke time bas har class ke andar, har feature ka mean aur variance nikaal lo — bas itna hi training hai! Phir jab naya point aata hai, to poochte ho: "Kaunsi class ki bell-curve ne is value ko sabse zyada easily produce kiya hoga?" Jis class ke liye point sabse kam surprising hai, wahi answer hai.

"Naive" isliye kehte hain kyunki hum assume karte hain ki saare features ek-dusre se independent hain (given the class). Real life mein ye galat hota hai, par phir bhi kaam karta hai — kyunki humein sirf sahi class choose karni hai, exact probability nahi chahiye. Is assumption se joint distribution tut ke chhote-chhote 1-D Gaussians ka product ban jaata hai, jo estimate karna aasan hai.

Ek important baat (Example 2 dekho): sirf "nearest mean" wala rule tabhi chalta hai jab sab classes ka variance aur prior barabar ho. Agar koi class bahut tight (chhota sigma) hai, to usse thoda door wala point bhi bahut unlikely ho jaata hai — isliye decision boundary curved (quadratic) banti hai. Aur practical tip: probabilities ko multiply mat karo, log leke add karo, warna numbers itne chhote ho jaate hain ki computer 0 bana deta hai (underflow).

Test yourself — SVM, Naive Bayes & Probabilistic Models

Connections