Hyperparameters C and gamma — AI-ML notes

WHY do these hyperparameters exist?

A hard-margin SVM assumes data is perfectly separable — no point is ever on the wrong side. Real data is noisy, so we allow violations. But if we allow violations for free, the model has no reason to fit anything. So we need a knob that prices those violations: that knob is C.

Separately, a linear boundary often isn't enough. We map data into a higher-dimensional space using a kernel. The most common is the RBF (Gaussian) kernel, whose "width" — how quickly similarity decays with distance — is set by gamma.

The soft-margin objective (deriving where C lives)

Derivation from scratch.

Start with the hard-margin idea: maximize the margin $=\dfrac{2}{\lVert w\rVert}$ , i.e. minimize $\tfrac12\lVert w\rVert^2$ , subject to every point being correctly classified with margin: $y_i(w^\top x_i + b) \ge 1.$

Why $\ge 1$ ? We fix the scale so the closest points sit exactly at $\pm 1$ ; the geometric margin then equals $1/\lVert w\rVert$ per side.

Real data breaks $y_i(w^\top x_i+b)\ge 1$ . So relax it by borrowing a nonnegative amount $\xi_i$ : $y_i(w^\top x_i + b) \ge 1 - \xi_i, \qquad \xi_i \ge 0.$

Why? $\xi_i=0$ means the point obeys the margin. $0<\xi_i<1$ means it's inside the margin but still correct. $\xi_i>1$ means it's misclassified.

We don't want free slack, so add its total cost to the objective:

Here $C>0$ is the penalty per unit of slack.

Large C → slack is expensive → the optimizer drives $\xi_i\to 0$ → boundary bends to classify training points correctly → low bias, high variance (risk of overfitting).
Small C → slack is cheap → the optimizer prefers a big margin (small $\lVert w\rVert^2$ ) even if some points are violated → smoother boundary, high bias (risk of underfitting).

Where gamma lives (deriving the RBF kernel)

Why this form? We want a similarity that is $1$ when points coincide and decays smoothly to $0$ as they separate. A Gaussian bump does exactly this. Comparing to the classic Gaussian $\exp\!\big(-\lVert x-x'\rVert^2/(2\sigma^2)\big)$ , we read off $\gamma = \frac{1}{2\sigma^2}.$

Interpretation of the derivation:

Large gamma ⇒ small $\sigma$ ⇒ the bump is narrow ⇒ each point only influences its very close neighbors ⇒ the decision surface becomes a set of tight islands around individual points → high variance / overfitting.
Small gamma ⇒ large $\sigma$ ⇒ the bump is wide ⇒ every point influences far away ⇒ the surface is smooth, almost linear → high bias / underfitting.

Forecast-then-Verify

Worked examples

Common mistakes (Steel-manned)

Recall Feynman: explain to a 12-year-old

Imagine drawing a line between red and blue marbles. C is how angry you get when a marble ends up on the wrong side. Very angry (big C) = you bend the line into weird shapes just to catch every stray marble — even the ones that rolled there by accident. Chill (small C) = you draw one neat line and shrug off a few strays. Gamma is how "loud" each marble shouts. Loud shout with short range (big gamma) = each marble only affects the spot right next to it, so you get lots of tiny bubbles around each marble. Quiet, far-reaching (small gamma) = marbles blend together and you get one smooth line. Best model = not too angry, not too loud.

Flashcards

What does C penalize in the soft-margin SVM?

The total slack

C\sum_i\xi_i

— i.e. the cost of margin violations/misclassifications.

Effect of large C on bias/variance?

Low bias, high variance (boundary fits training points tightly → overfitting risk).

Effect of small C?

Large margin, high bias, smoother boundary → underfitting risk.

Write the RBF kernel.

K(x,x')=\exp(-\gamma\lVert x-x'\rVert^2)

Relation between gamma and Gaussian width sigma?

\gamma = 1/(2\sigma^2)

; large gamma = small sigma = narrow bump.

At what distance does an RBF point's influence fall to 1/e?

r = 1/\sqrt{\gamma}

Large gamma effect?

Narrow influence, tight islands around points → high variance / overfitting.

Small gamma effect?

Wide influence, near-linear smooth boundary → high bias / underfitting.

Why must you standardize features before RBF-SVM?

Because the kernel uses

\lVert x-x'\rVert^2

; unscaled features make distances (and thus effective gamma) meaningless.

Should C and gamma be tuned separately?

No — jointly on a log-scaled 2D grid via cross-validation, since their optima are coupled.

Meaning of

\xi_i>1

The point is actually misclassified (past the boundary), not just inside the margin.

Constraint that introduces slack?

y_i(w^\top x_i+b)\ge 1-\xi_i,\ \xi_i\ge0

Connections

Soft-Margin SVM — where C originates in the primal objective.
Kernel Trick — why we need gamma at all.
RBF Kernel — the Gaussian kernel gamma parameterizes.
Bias-Variance Tradeoff — the lens for reading both knobs.
Cross-Validation — how to actually pick C and gamma.
Feature Scaling / Standardization — prerequisite for RBF.
Support Vectors — points with $\xi_i>0$ or on the margin.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, RBF-SVM me do main knobs hote hain: C aur gamma. C ka matlab hai "galti ki keemat" — agar training me koi point galat side pe chala jaata hai to usko kitna punish karein. Bada C = strict teacher, har galti pe bahut gussa, isliye boundary tedhi-medhi ho ke har point ko pakadne lagti hai (overfitting). Chhota C = chill teacher, thodi galti chalti hai, isliye ek smooth aur simple line banti hai (kabhi-kabhi underfit).

Gamma RBF kernel ki width set karta hai — ek point ka asar kitni door tak jaata hai. Formula yaad rakho: reach $= 1/\sqrt{\gamma}$ . Matlab gamma bada to reach chhota — har point sirf apne bilkul paas walon ko affect karta hai, isliye chhote-chhote islands ban jaate hain (overfit). Gamma chhota to reach bada — sab points ek doosre me ghul-mil jaate hain, boundary almost seedhi ho jaati hai (underfit).

Ek important baat: RBF kernel distance $\lVert x-x'\rVert^2$ use karta hai, isliye features ko pehle standardize karna zaroori hai. Warna gamma ka effective matlab hi bigad jaata hai. Aur C aur gamma dono ko saath me tune karo — log-scale grid pe cross-validation se — kyunki dono ka best value ek doosre pe depend karta hai.

Simple mantra: dono chhote = smooth/underfit, dono bade = bendy/overfit. Sweet spot beech me hota hai, jo validation accuracy se milta hai. Yehi 80/20 hai — bas ye directions samajh lo, baaki grid search kar lega.

Test yourself — SVM, Naive Bayes & Probabilistic Models

Connections