Hyperparameters C and gamma — Hinglish notes

2.4.5 · AI-ML › SVM, Naive Bayes & Probabilistic Models

YE hyperparameters exist kyun karte hain?

Ek hard-margin SVM assume karta hai ki data perfectly separable hai — koi bhi point kabhi wrong side pe nahi hoga. Real data noisy hota hai, isliye hum violations allow karte hain. Lekin agar hum violations free mein allow karein, toh model ke paas kuch bhi fit karne ki koi wajah nahi hogi. Toh humein ek knob chahiye jo un violations ko price kare: woh knob hai C.

Alag se, ek linear boundary aksar kaafi nahi hoti. Hum data ko ek higher-dimensional space mein map karte hain ek kernel use karke. Sabse common hai RBF (Gaussian) kernel, jiski "width" — ki similarity distance ke saath kitni jaldi decay hoti hai — gamma se set hoti hai.

Soft-margin objective (deriving karna ki C kahan rehta hai)

Scratch se Derivation.

Hard-margin idea se shuru karo: margin $= \frac{2}{∥ w ∥}$ maximize karo, yaani $\frac{1}{2} ∥ w ∥^{2}$ minimize karo, subject to ye ki har point margin ke saath correctly classify ho: $y_{i} (w^{⊤} x_{i} + b) \geq 1.$

$\geq 1$ kyun? Hum scale fix karte hain taaki closest points exactly $\pm 1$ pe baith jaayein; geometric margin tab $1/ ∥ w ∥$ per side ke barabar hoti hai.

Real data $y_{i} (w^{⊤} x_{i} + b) \geq 1$ break karta hai. Toh isse relax karo ek nonnegative amount $ξ_{i}$ borrow karke: $y_{i} (w^{⊤} x_{i} + b) \geq 1 - ξ_{i}, ξ_{i} \geq 0.$

Kyun? $ξ_{i} = 0$ matlab point margin ko obey karta hai. $0 < ξ_{i} < 1$ matlab woh margin ke andar hai lekin phir bhi correct hai. $ξ_{i} > 1$ matlab woh misclassified hai.

Hum free slack nahi chahte, toh uska total cost objective mein add karo:

Yahan $C > 0$ hai penalty per unit of slack.

Large C → slack expensive hai → optimizer $ξ_{i} \to 0$ drive karta hai → boundary training points ko correctly classify karne ke liye bend hoti hai → low bias, high variance (overfitting ka risk).
Small C → slack sasta hai → optimizer bada margin prefer karta hai (chhota $∥ w ∥^{2}$ ) chahe kuch points violate hon → smoother boundary, high bias (underfitting ka risk).

Gamma kahan rehta hai (RBF kernel derive karna)

Ye form kyun? Hum chahte hain ek similarity jo $1$ ho jab points coincide karein aur smoothly $0$ ki taraf decay ho jab woh alag hon. Ek Gaussian bump exactly yahi karta hai. Classic Gaussian $exp (- ∥ x - x^{'} ∥^{2} / (2 σ^{2}))$ se compare karne par hum paate hain $γ = \frac{1}{2 σ ^{2}} .$

Derivation ki interpretation:

Large gamma ⇒ chhota $σ$ ⇒ bump narrow hai ⇒ har point sirf apne bahut close neighbors ko influence karta hai ⇒ decision surface individual points ke aas-paas tight islands ka ek set ban jaati hai → high variance / overfitting.
Small gamma ⇒ bada $σ$ ⇒ bump wide hai ⇒ har point door tak influence karta hai ⇒ surface smooth, almost linear hai → high bias / underfitting.

Forecast-then-Verify

Worked examples

Common mistakes (Steel-manned)

Recall Feynman: 12-saal ke bacche ko explain karo

Socho tum lal aur neele marbles ke beech ek line kheench rahe ho. C hai ki tum kitna gussa hote ho jab ek marble wrong side pe pahunch jaati hai. Bahut gussa (bada C) = tum line ko ajeeb shapes mein mod lete ho sirf har stray marble pakadne ke liye — woh bhi jo accident se wahan pahunchi. Chill (chhota C) = tum ek neat line kheenchte ho aur kuch strays ko ignore kar dete ho. Gamma hai ki har marble kitna "zor se chillati" hai. Loud shout with short range (bada gamma) = har marble sirf apne barabar waali jagah ko affect karti hai, toh tumhe har marble ke aas-paas bahut saari tiny bubbles milti hain. Quiet, far-reaching (chhota gamma) = marbles blend ho jaati hain aur tumhe ek smooth line milti hai. Best model = na bahut gussa, na bahut loud.

Flashcards

Soft-margin SVM mein C kya penalize karta hai?

Total slack

C \sum_{i} ξ_{i}

— yaani margin violations/misclassifications ki cost.

Large C ka bias/variance par kya effect hota hai?

Low bias, high variance (boundary training points ko tightly fit karti hai → overfitting ka risk).

Small C ka kya effect hota hai?

Bada margin, high bias, smoother boundary → underfitting ka risk.

RBF kernel likho.

K (x, x^{'}) = exp (- γ ∥ x - x^{'} ∥^{2})

Gamma aur Gaussian width sigma mein kya relation hai?

γ = 1/ (2 σ^{2})

; bada gamma = chhota sigma = narrow bump.

Kis distance par ek RBF point ka influence 1/e tak girta hai?

r = 1/ γ

Large gamma ka kya effect hota hai?

Narrow influence, points ke aas-paas tight islands → high variance / overfitting.

Small gamma ka kya effect hota hai?

Wide influence, near-linear smooth boundary → high bias / underfitting.

RBF-SVM se pehle features standardize kyun karne chahiye?

Kyunki kernel

∥ x - x^{'} ∥^{2}

use karta hai; unscaled features distances (aur isliye effective gamma) ko meaningless bana dete hain.

Kya C aur gamma alag-alag tune karne chahiye?

Nahi — jointly ek log-scaled 2D grid par cross-validation ke through, kyunki unke optima coupled hain.

ξ_{i} > 1

ka kya matlab hai?

Point actually misclassified hai (boundary ke past), sirf margin ke andar nahi.

Slack introduce karne wala constraint kya hai?

y_{i} (w^{⊤} x_{i} + b) \geq 1 - ξ_{i}, ξ_{i} \geq 0

Connections

Soft-Margin SVM — jahan C primal objective mein originate hota hai.
Kernel Trick — isliye humein gamma ki zaroorat hai.
RBF Kernel — woh Gaussian kernel jise gamma parameterize karta hai.
Bias-Variance Tradeoff — dono knobs padhne ka lens.
Cross-Validation — actually C aur gamma kaise choose karein.
Feature Scaling / Standardization — RBF ke liye prerequisite.
Support Vectors — woh points jinke liye $ξ_{i} > 0$ ya margin par hain.