Linear, polynomial, and RBF kernels — Hinglish notes

2.4.4 · AI-ML › SVM, Naive Bayes & Probabilistic Models

Kernels ki zaroorat KYUN hai?

Linear SVM KYA karta hai: yeh ek separating hyperplane $w^{⊤} x + b = 0$ dhundhta hai. Iska decision function, training ke baad, sirf dot products ke zariye likha ja sakta hai — query point $x$ aur support vectors $x_{i}$ ke beech:

$f (x) = \sum_{i} α_{i} y_{i} (x_{i}^{⊤} x) + b$

Yeh KYUN matter karta hai: data $x$ sirf dot product $x_{i}^{⊤} x$ ke andar aata hai. Toh agar data apni raw form mein linearly separable nahi hai, toh hum ise $ϕ (x)$ se ek richer space mein map kar sakte hain aur wahan separable hone ki umeed rakh sakte hain. Phir decision function ko $ϕ (x_{i})^{⊤} ϕ (x)$ chahiye hoga.

Trick humein KAISE bachati hai: $ϕ (x)$ explicitly compute karna expensive ya infinite-dimensional ho sakta hai. Lekin agar ek function $K$ ho jisme

$K (x, z) = ϕ (x)^{⊤} ϕ (z)$

toh hum $ϕ$ kabhi nahi banate. Bas har dot product ko $K$ se replace kar dete hain. Yahi poora game hai.

Teen workhorse kernels

Yahan $γ$ (gamma), $c$ (coef0), aur $d$ (degree) hyperparameters hain. Chaliye har ek ka feature map derive karte hain taki samajh aaye yeh valid KYUN hain.

1. Linear kernel — trivial map

Derivation. $ϕ (x) = x$ lo. Phir $ϕ (x)^{⊤} ϕ (z) = x^{⊤} z = K (x, z)$ . Ho gaya. Yeh identity feature map ke saath ek kernel hai. Iska use tab karo jab $d$ bada ho aur data already ~linearly separable ho (text, sparse features).

2. Polynomial kernel — feature map scratch se derive karo

Sabse simple case lo: $d = 2$ , $c = 1$ , $γ = 1$ , 2D mein $x = (x_{1}, x_{2})$ , $z = (z_{1}, z_{2})$ .

$K (x, z) = (x^{⊤} z + 1)^{2} = (x_{1} z_{1} + x_{2} z_{2} + 1)^{2}$

Ise expand karo (yahi kyun yeh kernel hai — hum $ϕ$ padh sakte hain):

$= x_{1}^{2} z_{1}^{2} + x_{2}^{2} z_{2}^{2} + 2 x_{1} x_{2} z_{1} z_{2} + 2 x_{1} z_{1} + 2 x_{2} z_{2} + 1$

Yeh step KYUN? Har term ko (kuch $x$ mein) $\times$ (wahi kuch $z$ mein) ke roop mein group karo. Yahi grouping ek dot product $ϕ (x)^{⊤} ϕ (z)$ hai jisme

$ϕ (x) = (x_{1}^{2}, x_{2}^{2}, 2 x_{1} x_{2}, 2 x_{1}, 2 x_{2}, 1)$

Toh degree-2 polynomial kernel chupke se aapko saare squares, cross-terms aur linear terms deta hai — 2 inputs se ek 6-dimensional feature space, sirf ek dot product aur ek square se compute kiya gaya. $c$ control karta hai ki low-order terms ko kitna weight mile; bada $c$ → lower-degree features par zyada emphasis.

3. RBF kernel — yeh infinite-dimensional map KYUN hai

$∥ x - z ∥^{2} = x^{⊤} x - 2 x^{⊤} z + z^{⊤} z$ se shuru karo:

$K (x, z) = e^{- γ ∥ x - z ∥^{2}} = normalizers e^{- γ ∥ x ∥^{2}} e^{- γ ∥ z ∥^{2}} e^{2 γ x^{⊤} z}$

Ab last factor ko Taylor-expand karo:

$e^{2 γ x^{⊤} z} = \sum_{k = 0}^{\infty} \frac{( 2 γ ) ^{k}}{k !} (x^{⊤} z)^{k}$

Yeh step KYUN? Har $(x^{⊤} z)^{k}$ khud degree $k$ ka ek (homogeneous) polynomial kernel hai, yaani ek finite space mein dot product. Kernels ka weighted sum bhi ek kernel hota hai, aur yahan sum infinity tak jaata hai → RBF ek infinite-dimensional feature map se correspond karta hai jisme har degree ke polynomial features hain. Isliye yeh itna expressive hai.

Figure — Linear, polynomial, and RBF kernels

Worked examples

Common mistakes (steel-manned)

Active recall

Recall Khud test karo (answers chhupao)

SVM ko sirf $K$ kyun chahiye, $ϕ$ kyun nahi? → Decision function mein data sirf dot products ke andar aata hai.
Valid kernel ke liye Gram matrix mein kya property honi chahiye? → Symmetric positive semi-definite (Mercer).
RBF "infinite dimensional" KYUN hai? → $e^{2 γ x^{⊤} z}$ ka Taylor expansion polynomial kernels ka ek infinite weighted sum hai.
RBF mein $γ$ kya control karta hai? → Har point ki reach/locality; bias–variance tradeoff.

Flashcards

Kernel trick kya hai?

Har dot product

x^{⊤} z

ko algorithm mein

K (x, z) = ϕ (x)^{⊤} ϕ (z)

se replace karna, ek high-dim feature space gain karna bina

ϕ

compute kiye.

Linear kernel ka formula?

K (x, z) = x^{⊤} z

, feature map

ϕ (x) = x

ke saath.

Polynomial kernel ka formula?

K (x, z) = (γ x^{⊤} z + c)^{d}

RBF/Gaussian kernel ka formula?

K (x, z) = exp (- γ ∥ x - z ∥^{2})

Kaunsi condition kisi function ko valid kernel banati hai?

Iska Gram matrix symmetric positive semi-definite hona chahiye (Mercer's theorem).

2D mein degree-2 poly kernel

(x^{⊤} z + 1)^{2}

ka feature map?

(x_{1}^{2}, x_{2}^{2}, 2 x_{1} x_{2}, 2 x_{1}, 2 x_{2}, 1)

RBF infinite-dimensional KYUN hai?

Kyunki

e^{2 γ x^{⊤} z} = \sum_{k} \frac{( 2 γ ) ^{k}}{k !} (x^{⊤} z)^{k}

har degree ke polynomial kernels ko mix karta hai.

RBF mein bade

γ

ka effect?

Bahut local influence → wiggly boundary → overfitting ka risk.

RBF mein chote

γ

ka effect?

Smooth, near-linear boundary → underfitting ka risk.

Linear kernel kab prefer karo?

High-dimensional sparse data (text) jahan nonlinearity overfit kare; speed ke liye bhi.

RBF/poly se pehle features scale KYUN karne chahiye?

Yeh

∥ x - z ∥^{2}

x^{⊤} z

par depend karte hain, jo large-magnitude features se dominate hote hain.

Recall Feynman: ek 12-saal ke bachche ko samjhao

Socho tum do khilaunon ki "alikeness" judge kar rahe ho. Linear tarika: unhe line up karo aur dekho kitna same direction mein point karte hain. Polynomial tarika: unke combinations bhi compare karo — colour AUR size saath mein, sirf akela nahi. RBF tarika: bas measure karo kitne paas hain — touching khilaune "1" hain, door wale "0" hain. Jaadu ki baat yeh hai ki computer yeh alikeness seedha measure kar sakta hai, bina har possible combination ki giant table banaye. Yeh shortcut SVM ko groups ke beech curved fences draw karne deta hai, jabki sirf simple arithmetic karta hai.

Connections

Support Vector Machines — kernels dual decision function mein plug hote hain.
The Kernel Trick — teeno ke peeche ka general principle.
Mercer's Theorem — PSD Gram matrix condition.
Bias-Variance Tradeoff — $γ$ , $d$ , aur $C$ ise control karte hain.
Feature Scaling / Standardization — required preprocessing.
Overfitting and Regularization — complexity kyun nahi badhani chahiye.
Gaussian / RBF functions — Naive Bayes densities jaisi same bell shape.