Hard vs soft margin classifiers — AI-ML notes

WHY do we need margins at all?

The decision boundary is a hyperplane: $\mathbf{w}^\top \mathbf{x} + b = 0$ We predict class $+1$ if $\mathbf{w}^\top\mathbf{x}+b > 0$ , class $-1$ otherwise.

Deriving the margin from scratch

WHAT is the distance from a point $\mathbf{x}_0$ to the hyperplane $\mathbf{w}^\top\mathbf{x}+b=0$ ?

Why this step? Geometry: move from $\mathbf{x}_0$ along the unit normal $\mathbf{w}/\|\mathbf{w}\|$ until you hit the plane. The signed distance is $d = \frac{\mathbf{w}^\top \mathbf{x}_0 + b}{\|\mathbf{w}\|}.$

We have freedom to scale $\mathbf{w}$ and $b$ by any constant — the hyperplane is unchanged. So we fix the scale by demanding that the closest points satisfy $|\mathbf{w}^\top\mathbf{x}+b| = 1$ . This is the canonical form.

Why this step? It removes the ambiguity so the optimization is well-posed. With this choice the margin (distance to the nearest point) becomes $\gamma = \frac{1}{\|\mathbf{w}\|}, \qquad \text{full gap width} = \frac{2}{\|\mathbf{w}\|}.$

So maximizing the margin $=$ minimizing $\|\mathbf{w}\|$ $=$ minimizing $\frac{1}{2}\|\mathbf{w}\|^2$ (the square and $\tfrac12$ are just for a nicer, convex, differentiable objective).

Hard margin SVM

WHY the constraint $y_i(\mathbf{w}^\top\mathbf{x}_i+b)\ge 1$ ?

For a $+1$ point we want $\mathbf{w}^\top\mathbf{x}_i+b \ge 1$ (well on the positive side).
For a $-1$ point we want $\mathbf{w}^\top\mathbf{x}_i+b \le -1$ .
Multiplying by $y_i\in\{+1,-1\}$ folds both into the single clean inequality $y_i(\dots)\ge 1$ .

Soft margin SVM

We introduce a slack variable $\xi_i \ge 0$ per point: how far point $i$ is allowed to violate its margin constraint.

$y_i(\mathbf{w}^\top\mathbf{x}_i + b) \ge 1 - \xi_i, \qquad \xi_i \ge 0.$

HOW to read $\xi_i$ :

$\xi_i = 0$ : point is safely outside the margin (obeys hard constraint).
$0 < \xi_i \le 1$ : inside the margin but still correctly classified.
$\xi_i > 1$ : misclassified (on the wrong side of the boundary).

We don't want unlimited cheating, so we penalize the total slack:

WHY the $C$ ? $C$ is the trade-off knob between a wide margin and few violations.

Large $C$ → violations are very expensive → model tries hard to classify all points → narrow margin, low bias, high variance → approaches hard margin as $C\to\infty$ .
Small $C$ → cheap to violate → wide margin, more tolerance to noise, high bias, low variance.

Figure — Hard vs soft margin classifiers

Hinge loss view (WHY it's the same thing)

At the optimum, each $\xi_i$ is as small as allowed. The constraints force $\xi_i = \max\bigl(0,\ 1 - y_i(\mathbf{w}^\top\mathbf{x}_i+b)\bigr).$

Why this step? If the point already satisfies $y_i(\dots)\ge 1$ , no slack needed → $\xi_i=0$ . Otherwise $\xi_i$ must cover the gap exactly. Substituting turns the constrained problem into an unconstrained one:

Worked examples

Support vectors (bonus insight)

Only points with $\xi_i>0$ or lying exactly on the margin ( $y_i(\dots)=1$ ) have nonzero influence — these are the support vectors. Points comfortably outside the margin can be deleted without changing the boundary. WHY? In the dual, their Lagrange multipliers $\alpha_i=0$ .

Recall Feynman: explain to a 12-year-old

Imagine drawing a fence between the cats and the dogs in a park. A hard fence rule says: no animal may touch or cross the fence, ever. That's fine until one confused cat wanders into dog-land — now you can't draw any fence at all! The soft rule says: try to keep them apart with the widest safe path, but if a few troublemakers cross, that's okay — just count how badly they cross and try to keep that small. The knob $C$ is how angry you get at troublemakers: big $C$ = very strict, small $C$ = chill and forgiving.

Flashcards

What is the margin width of an SVM in terms of w?

\frac{2}{\|\mathbf{w}\|}

(half-margin

1/\|\mathbf{w}\|

to each side).

Why does maximizing margin equal minimizing

\|\mathbf{w}\|

With canonical scaling

|\mathbf{w}^\top x+b|=1

at closest points, margin

=1/\|\mathbf{w}\|

, so wider margin means smaller norm. ::: We minimize

\tfrac12\|\mathbf{w}\|^2

for convexity/differentiability.

Write the hard margin constraint.

y_i(\mathbf{w}^\top\mathbf{x}_i+b)\ge 1

for all

i

When does the hard margin SVM have NO solution?

When the data is not linearly separable (constraints are infeasible).

What is a slack variable

\xi_i

A nonnegative amount by which point

i

is allowed to violate its margin constraint:

y_i(\mathbf{w}^\top x_i+b)\ge 1-\xi_i

Meaning of

\xi_i=0

0<\xi_i\le1

\xi_i>1

0 = outside margin (safe); (0,1] = inside margin but correct; >1 = misclassified.

Write the soft margin primal objective.

\min \tfrac12\|\mathbf{w}\|^2 + C\sum_i \xi_i

s.t.

y_i(\mathbf{w}^\top x_i+b)\ge 1-\xi_i,\ \xi_i\ge0

Soft margin as unconstrained loss?

\min \tfrac12\|\mathbf{w}\|^2 + C\sum_i \max(0,\,1-y_i(\mathbf{w}^\top x_i+b))

(regularized hinge loss).

What does the parameter

C

trade off?

Margin width vs. training violations. Large C = narrow margin/overfit; small C = wide margin/more tolerant.

What happens to soft margin as

C\to\infty

It reduces to the hard margin SVM (violations become infinitely costly).

What is a support vector?

A point on the margin or violating it (

\xi_i>0

); it has nonzero Lagrange multiplier and defines the boundary.

Formula for slack at the optimum?

\xi_i=\max(0,\,1-y_i(\mathbf{w}^\top x_i+b))

(the hinge).

Connections

Support Vector Machines — the parent method.
Kernel Trick — how soft margin extends to nonlinear boundaries.
Hinge Loss — the equivalent unconstrained objective.
Regularization and Bias-Variance Tradeoff — $C$ as inverse regularization strength.
Lagrangian Duality and KKT Conditions — where support vectors come from.
Logistic Regression — contrast: log-loss vs hinge-loss classifiers.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, SVM ka main idea ye hai ki do classes ke beech me hum aisi line (ya hyperplane) khichte hain jo dono ke sabse pass wale points se maximum door ho. Is gap ko margin bolte hain, aur wo $2/\|\mathbf{w}\|$ hota hai. To margin bada karna matlab $\|\mathbf{w}\|$ chhota karna — isliye hum $\tfrac12\|\mathbf{w}\|^2$ minimize karte hain. Wide margin zyada robust hota hai, thoda sa noise aane par bhi prediction nahi badalti.

Hard margin kehta hai: koi bhi point margin ke andar ya galat side me nahi aana chahiye — 100% perfect separation. Problem ye hai ki agar data thoda bhi overlap kare ya ek bhi outlier ho, to koi solution hi nahi banta, aur model overfit ho jaata hai. Isliye real duniya me hard margin kaam nahi karta.

Yahan aata hai soft margin. Har point ke liye ek slack variable $\xi_i$ dete hain — matlab "thoda cheating allowed hai". $\xi_i=0$ matlab point safe hai, $0<\xi_i\le1$ matlab margin ke andar par sahi side, aur $\xi_i>1$ matlab galat classify ho gaya. Total cheating ko penalize karte hain $C\sum\xi_i$ se. Ye pura cheez hinge loss ke barabar hai: $\max(0, 1-y_i(\mathbf{w}^\top x_i+b))$ .

$C$ ek knob hai. Bada $C$ = strict teacher, violations ko bahut mehnga maanega, narrow margin, overfit. Chhota $C$ = chill teacher, wide margin, noise ko ignore karega, better generalization. Jab $C\to\infty$ , soft margin wapas hard margin ban jaata hai. Exam me yaad rakho: "Hard = zero tolerance, Soft = slack with a Cost."

Test yourself — SVM, Naive Bayes & Probabilistic Models

Connections