Hard vs soft margin classifiers — Hinglish notes

2.4.2 · AI-ML › SVM, Naive Bayes & Probabilistic Models

Margins ki zaroorat kyun hai?

Decision boundary ek hyperplane hai: $w^{⊤} x + b = 0$ Hum class $+ 1$ predict karte hain agar $w^{⊤} x + b > 0$ , warna class $- 1$ .

Margin ko scratch se derive karna

KYA hai distance ek point $x_{0}$ se hyperplane $w^{⊤} x + b = 0$ tak?

Yeh step kyun? Geometry: $x_{0}$ se unit normal $w /∥ w ∥$ ke along move karo jab tak plane hit na ho. Signed distance hai $d = \frac{w ^{⊤} x _{0} + b}{∥ w ∥} .$

Hamare paas freedom hai ki $w$ aur $b$ ko kisi bhi constant se scale karein — hyperplane unchanged rehta hai. Toh hum scale fix karte hain yeh demand karke ki closest points satisfy karein $∣ w^{⊤} x + b ∣ = 1$ . Yahi canonical form hai.

Yeh step kyun? Yeh ambiguity remove karta hai taaki optimization well-posed ho. Is choice ke saath margin (nearest point tak distance) ban jaata hai $γ = \frac{1}{∥ w ∥}, full gap width = \frac{2}{∥ w ∥} .$

Toh margin maximize karna $=$ $∥ w ∥$ minimize karna $=$ $\frac{1}{2} ∥ w ∥^{2}$ minimize karna (square aur $\frac{1}{2}$ sirf ek nicer, convex, differentiable objective ke liye hain).

Hard margin SVM

Constraint $y_{i} (w^{⊤} x_{i} + b) \geq 1$ kyun?

Ek $+ 1$ point ke liye hum chahte hain $w^{⊤} x_{i} + b \geq 1$ (positive side par acchi tarah se).
Ek $- 1$ point ke liye hum chahte hain $w^{⊤} x_{i} + b \leq - 1$ .
$y_{i} \in {+ 1, - 1}$ se multiply karna dono ko ek single clean inequality $y_{i} (\dots) \geq 1$ mein fold kar deta hai.

Soft margin SVM

Hum har point ke liye ek slack variable $ξ_{i} \geq 0$ introduce karte hain: point $i$ ko apni margin constraint violate karne ki kitni permission hai.

$y_{i} (w^{⊤} x_{i} + b) \geq 1 - ξ_{i}, ξ_{i} \geq 0.$

$ξ_{i}$ ko kaise padhein:

$ξ_{i} = 0$ : point safely margin ke bahar hai (hard constraint maanta hai).
$0 < ξ_{i} \leq 1$ : margin ke andar hai lekin phir bhi correctly classified hai.
$ξ_{i} > 1$ : misclassified (boundary ke galat side par).

Hum unlimited cheating nahi chahte, toh hum total slack ko penalize karte hain:

$C$ kyun? $C$ ek trade-off knob hai wide margin aur kam violations ke beech.

Large $C$ → violations bahut expensive hain → model saare points classify karne ki koshish karta hai → narrow margin, low bias, high variance → $C \to \infty$ par hard margin ke paas pahunch jaata hai.
Small $C$ → violate karna sasta hai → wide margin, noise ke liye zyada tolerance, high bias, low variance.

Figure — Hard vs soft margin classifiers

Hinge loss view (kyun yeh same cheez hai)

Optimum par, har $ξ_{i}$ utna hi chota hota hai jitna allow hai. Constraints force karte hain $ξ_{i} = max (0, 1 - y_{i} (w^{⊤} x_{i} + b)) .$

Yeh step kyun? Agar point already $y_{i} (\dots) \geq 1$ satisfy karta hai, koi slack nahi chahiye → $ξ_{i} = 0$ . Warna $ξ_{i}$ ko gap exactly cover karna hoga. Substitute karne par constrained problem ek unconstrained mein convert ho jaata hai:

Worked examples

Support vectors (bonus insight)

Sirf woh points jinke liye $ξ_{i} > 0$ ya jo exactly margin par hain ( $y_{i} (\dots) = 1$ ) ka nonzero influence hota hai — yahi support vectors hain. Points jo margin ke bahar comfortably hain unhe delete kiya ja sakta hai bina boundary change kiye. Kyun? Dual mein unke Lagrange multipliers $α_{i} = 0$ hote hain.

Recall Feynman: ek 12-saal ke bacche ko explain karo

Socho tum ek park mein cats aur dogs ke beech fence kheench rahe ho. Hard fence rule kehta hai: koi bhi jaanwar fence ko chhu ya cross nahi kar sakta, kabhi bhi. Yeh theek hai jab tak ek confused cat dog-land mein nahi chali jaati — ab tum koi bhi fence nahi kheench sakte! Soft rule kehta hai: unhe sabse wide safe path ke saath alag rakhne ki koshish karo, lekin agar kuch troublemakers cross kar jaayein, koi baat nahi — bas count karo ki woh kitna cross karte hain aur use chota rakhne ki koshish karo. Knob $C$ yeh hai ki tum troublemakers par kitne naraaz ho: bada $C$ = bahut strict, chota $C$ = chill aur maafi dene wala.

Flashcards

What is the margin width of an SVM in terms of w?

\frac{2}{∥ w ∥}

(half-margin

1/∥ w ∥

har side ko).

Why does maximizing margin equal minimizing

∥ w ∥

Canonical scaling

∣ w^{⊤} x + b ∣ = 1

closest points par, margin

= 1/∥ w ∥

, toh wider margin matlab smaller norm. ::: Hum

\frac{1}{2} ∥ w ∥^{2}

minimize karte hain convexity/differentiability ke liye.

Write the hard margin constraint.

y_{i} (w^{⊤} x_{i} + b) \geq 1

for all

i

When does the hard margin SVM have NO solution?

Jab data linearly separable nahi hota (constraints infeasible hote hain).

What is a slack variable

ξ_{i}

Ek nonnegative amount jitna point

i

ko apni margin constraint violate karne ki permission hai:

y_{i} (w^{⊤} x_{i} + b) \geq 1 - ξ_{i}

Meaning of

ξ_{i} = 0

0 < ξ_{i} \leq 1

ξ_{i} > 1

0 = margin ke bahar (safe); (0,1] = margin ke andar lekin correct; >1 = misclassified.

Write the soft margin primal objective.

min \frac{1}{2} ∥ w ∥^{2} + C \sum_{i} ξ_{i}

s.t.

y_{i} (w^{⊤} x_{i} + b) \geq 1 - ξ_{i}, ξ_{i} \geq 0

Soft margin as unconstrained loss?

min \frac{1}{2} ∥ w ∥^{2} + C \sum_{i} max (0, 1 - y_{i} (w^{⊤} x_{i} + b))

(regularized hinge loss).

What does the parameter

C

trade off?

Margin width vs. training violations. Large C = narrow margin/overfit; small C = wide margin/zyada tolerant.

What happens to soft margin as

C \to \infty

Yeh hard margin SVM mein reduce ho jaata hai (violations infinitely costly ho jaate hain).

What is a support vector?

Ek point jo margin par hai ya use violate kar raha hai (

ξ_{i} > 0

); iska nonzero Lagrange multiplier hota hai aur yeh boundary define karta hai.

Formula for slack at the optimum?

ξ_{i} = max (0, 1 - y_{i} (w^{⊤} x_{i} + b))

(the hinge).

Connections

Support Vector Machines — parent method.
Kernel Trick — soft margin nonlinear boundaries tak kaise extend hota hai.
Hinge Loss — equivalent unconstrained objective.
Regularization and Bias-Variance Tradeoff — $C$ as inverse regularization strength.
Lagrangian Duality and KKT Conditions — support vectors kahan se aate hain.
Logistic Regression — contrast: log-loss vs hinge-loss classifiers.