Gaussian Naive Bayes — Hinglish notes

2.4.8 · AI-ML › SVM, Naive Bayes & Probabilistic Models

YEH method exist kyun karti hai?

Hum $P (class ∣ features)$ chahte hain. Continuous data ke liye ise directly model karna mushkil hai. Bayes' theorem hume ise flip karne deta hai un chezon mein jo hum data se estimate kar sakte hain:

$P (y ∣ x) = \frac{P ( y ) P ( x ∣ y )}{P ( x )}$

KYA chahiye: posterior $P (y ∣ x)$ .
KYA measure kar sakte hain: prior $P (y)$ (classes count karo) aur likelihood $P (x ∣ y)$ (har class ke andar features kaise distributed hain).

Problem yeh hai: $P (x ∣ y) = P (x_{1}, x_{2}, \dots, x_{d} ∣ y)$ saari features par ek joint distribution hai — limited data se estimate karna bilkul impossible hai. "Naive" trick yahi fix karti hai.

Naive assumption (80/20 core)

YEH itna bada fayda kyun hai? $d$ features par ek joint distribution ke liye exponentially zyada parameters chahiye. Ise $d$ alag 1-D distributions mein todne par sirf $O (d)$ parameters lagte hain. Yeh "naive" hai kyunki features rarely truly independent hote hain — lekin classifier phir bhi surprisingly well kaam karta hai.

Gaussian part: har $P (x_{j} ∣ y)$ ko model karna

Hume sirf har feature har class ke liye do numbers chahiye: mean $μ_{y, j}$ aur variance $σ_{y, j}^{2}$ .

$μ$ aur $σ^{2}$ estimate kaise karte hain? (Scratch se derivation)

Hum Maximum Likelihood Estimation (MLE) use karte hain. Class $y$ mein saare $n_{y}$ training points lo; unki feature- $j$ values hain $x^{(1)}, \dots, x^{(n_{y})}$ . Gaussian ka log-likelihood hai:

$ℓ (μ, σ^{2}) = \sum_{i = 1}^{n_{y}} [- \frac{1}{2} ln (2 π σ^{2}) - \frac{( x ^{(i)} - μ ) ^{2}}{2 σ ^{2}}]$

$μ$ ke liye solve karo — $\partial ℓ / \partial μ = 0$ set karo: $\frac{\partial ℓ}{\partial μ} = \sum_{i} \frac{x ^{(i)} - μ}{σ ^{2}} = 0 \Rightarrow μ = \frac{1}{n _{y}} i \sum x^{(i)}$ Yeh step kyun? Variance cancel ho jaata hai, sirf sample mean bachta hai.

$σ^{2}$ ke liye solve karo — $\partial ℓ / \partial σ^{2} = 0$ set karo: $\frac{\partial ℓ}{\partial σ ^{2}} = \sum_{i} [- \frac{1}{2 σ ^{2}} + \frac{( x ^{(i)} - μ ) ^{2}}{2 σ ^{4}}] = 0 \Rightarrow σ^{2} = \frac{1}{n _{y}} i \sum (x^{(i)} - μ)^{2}$ Yeh step kyun? $2 σ^{4}$ se multiply karo aur rearrange karo — sample variance milta hai.

Toh training bas yeh hai: har class, har feature ka mean aur variance compute karo. Bas itna hi.

Decision rule (sab milaao)

Kyunki $P (x)$ har class ke liye same hai, hum ise drop karte hain aur numerator maximise karte hain:

$\overset{y}{^} = ar g max_{y} P (y) \prod_{j = 1}^{d} P (x_{j} ∣ y)$

Worked Example 1 — ek feature, do classes

Feature = height (cm). Class A (short): $μ_{A} = 160, σ_{A} = 5$ . Class B (tall): $μ_{B} = 180, σ_{B} = 5$ . Priors equal. $x = 172$ classify karo.

$ln P (x ∣ A) \propto - \frac{( 172 - 160 ) ^{2}}{2 \cdot 25} = - \frac{144}{50} = - 2.88$ Kyun? Constant $ln (2 π σ^{2})$ dono classes ke liye identical hai (same $σ$ ), toh cancel ho jaata hai.
$ln P (x ∣ B) \propto - \frac{( 172 - 180 ) ^{2}}{2 \cdot 25} = - \frac{64}{50} = - 1.28$

$- 1.28 > - 2.88 \Rightarrow$ Class B (tall). Kyun? $172$ , $180$ ke zyada paas hai; equal $σ$ aur equal priors ke saath, GNB bas nearest mean pick karta hai.

Worked Example 2 — unequal variance intuition ko flip karta hai

Class A: $μ_{A} = 160, σ_{A} = 20$ . Class B: $μ_{B} = 180, σ_{B} = 3$ . $x = 172$ classify karo.

$ln P (x ∣ A) = - \frac{1}{2} ln (2 π \cdot 400) - \frac{144}{800} = - 4.03 - 0.18 = - 4.21$
$ln P (x ∣ B) = - \frac{1}{2} ln (2 π \cdot 9) - \frac{64}{18} = - 2.02 - 3.56 = - 5.58$

$- 4.21 > - 5.58 \Rightarrow$ Class A, chahe $172$ B ke mean ke zyada paas ho! Yeh step kyun? B bahut "tight" hai ( $σ = 3$ ): uske liye 12 units door ki value extremely unlikely hai. $- \frac{1}{2} ln (2 π σ^{2})$ term aur denominator mein variance dono penalise karte hain confident-but-far predictions ko. Isliye GNB decision boundaries curved (quadratic) hoti hain, sirf midpoints nahi.

Worked Example 3 — do features

Point $x = (x_{1}, x_{2})$ . GNB score karta hai $ln P (y) - \frac{( x _{1} - μ _{y 1} ) ^{2}}{2 σ _{y 1}^{2}} - \frac{( x _{2} - μ _{y 2} ) ^{2}}{2 σ _{y 2}^{2}} + const_{y}$ . Kyun? Independence assumption joint ko per-feature log-terms ki ek sum mein badal deta hai — har feature alag vote karta hai.

Common mistake Common errors ko steel-man karna

(1) "Naive = features actually independent hain." Sahi lagta hai kyunki yahi assumption stated hai. Fix: yeh ek assumption hai jo hum impose karte hain, usually galat; GNB iske bawajood kaam karta hai kyunki argmax robust hai.

(2) "GNB nearest class mean pick karta hai." Sahi lagta hai aur sirf tab sach hai jab saare variances/priors equal hon (Example 1). Fix: unequal $σ$ ya priors boundary ko mod dete hain (Example 2). Variance aur $ln σ$ terms matter karte hain.

(3) "Raw probabilities multiply karo." Natural lagta hai. Fix: bahut saari features ke saath $0.0$ tak underflow ho jaata hai; hamesha logs sum karo.

(4) " $n - 1$ (unbiased) variance use karo." Statistically sahi lagta hai. Fix: MLE denominator mein $n$ deta hai; scikit-learn bhi ek tiny var_smoothing $σ^{2}$ mein add karta hai taaki divide-by-zero se bacha ja sake jab ek feature constant ho.

Recall Feynman: 12-saal ke bacche ko explain karo

Socho ki har class ek aisa insaan hai jo ek number line par darts fenkta hai, apni favourite jagah (mean) par aim karta hai. Kuch throwers kaanpte hain (bada spread) aur kuch super precise hain (chhota spread). Ek naya dart aata hai. Tum andaza lagate ho kaun sa thrower tha yeh pooch ke: "Har thrower ke liye, main kitna surprised hota agar UNHONE ise yahan pheka?" Tum us thrower ko choose karte ho jo least surprised hoga — yeh dekhte hue ki woh unke target ke kitna paas hai aur woh kitne precise hote hain. Precise thrower ko door ke darts par kam credit milta hai; kaanpne wale ko maafi milti hai. Yahi "least surprised" rule Gaussian Naive Bayes hai.

Flashcards

Naive Bayes ko "naive" banane wali core assumption kya hai?

Features class diye jaane par conditionally independent hain, isliye

P (x ∣ y) = \prod_{j} P (x_{j} ∣ y)

Naive assumption computationally kyun help karta hai?

Yeh ek joint distribution (exponential parameters) ko

d

alag 1-D distributions mein reduce karta hai (

O (d)

parameters).

Gaussian NB mein har

P (x_{j} ∣ y)

ko kaun si distribution model karti hai?

Ek univariate normal jisme class/feature-specific mean

μ_{y, j}

aur variance

σ_{y, j}^{2}

hota hai.

μ

aur

σ^{2}

ke MLE estimates kya hain?

μ = \frac{1}{n} \sum x_{i}

(sample mean) aur

σ^{2} = \frac{1}{n} \sum (x_{i} - μ)^{2}

(sample variance,

n

se divide).

Decision rule mein

P (x)

kyun drop karte hain?

Yeh saari classes ke liye same hai, isliye argmax nahi badalta.

Probabilities ke logs kyun lete hain?

Bahut saari chhoti numbers ke products underflow karte hain; log monotonic hai isliye argmax preserved rehta hai aur products sums ban jaate hain.

GNB "nearest class mean" mein kab reduce hota hai?

Jab saari classes mein equal variance aur equal priors hon.

GNB decision boundary generally quadratic kyun hoti hai?

(x - μ)^{2} /2 σ^{2}

aur

- \frac{1}{2} ln σ^{2}

terms har class ke liye alag hote hain, isliye ek quadratic (curved) boundary banti hai.

var_smoothing kis kaam aata hai?

Variances mein ek chhoti value add ki jaati hai taaki zero variance hone par division by zero se bacha ja sake.

Full GNB decision rule (log form)?

\overset{y}{^} = ar g max_{y} [ln P (y) + \sum_{j} (- \frac{1}{2} ln (2 π σ_{y, j}^{2}) - \frac{( x _{j} - μ _{y, j} ) ^{2}}{2 σ _{y, j}^{2}})]

Connections

Bayes Theorem — woh foundation jise GNB rewrite karta hai.
Naive Bayes — general family (continuous ke liye Gaussian, counts ke liye Multinomial Naive Bayes, binary ke liye Bernoulli Naive Bayes).
Maximum Likelihood Estimation — jisse $μ, σ^{2}$ derive hote hain.
Gaussian Distribution — likelihood model.
LDA & QDA — GNB essentially QDA hai diagonal covariance ke saath (independent features).
Logistic Regression — ek discriminative cousin jo $P (y ∣ x)$ directly model karta hai.
Log-Sum-Exp trick — stable posterior normalisation.