Support vectors interpretation — AI-ML notes

What is a support vector?

WHAT this says: the functional margin equals exactly $1$ for these points — they sit on the dashed "margin lines", not comfortably far away.

Deriving why only support vectors matter (from scratch)

We start from the hard-margin SVM primal problem. WHY this objective? We want the widest "street" between classes; the street half-width is $1/\|\mathbf{w}\|$ , so maximizing margin = minimizing $\|\mathbf{w}\|^2$ .

$\min_{\mathbf{w},b}\ \tfrac{1}{2}\|\mathbf{w}\|^2 \quad \text{s.t. } y_i(\mathbf{w}^\top\mathbf{x}_i + b) \ge 1\ \ \forall i.$

HOW we bring in the constraints: build the Lagrangian with multipliers $\alpha_i \ge 0$ :

$\mathcal{L}(\mathbf{w},b,\boldsymbol\alpha) = \tfrac12\|\mathbf{w}\|^2 - \sum_i \alpha_i\big[y_i(\mathbf{w}^\top\mathbf{x}_i + b) - 1\big].$

Why the minus sign? We subtract a penalty that grows when a constraint is violated, so the min–max saddle point enforces the constraints.

Stationarity — set derivatives to zero:

$\frac{\partial \mathcal L}{\partial \mathbf w}=0 \;\Rightarrow\; \mathbf{w} = \sum_i \alpha_i y_i \mathbf{x}_i.$

Why this step? This is the punchline: $\mathbf w$ is a weighted sum of the training points, weighted by $\alpha_i$ .

$\frac{\partial \mathcal L}{\partial b}=0 \;\Rightarrow\; \sum_i \alpha_i y_i = 0.$

Now the KKT complementary slackness condition (the heart of the interpretation):

$\boxed{\alpha_i\big[y_i(\mathbf{w}^\top\mathbf{x}_i + b) - 1\big] = 0.}$

HOW to read it: for each point either $\alpha_i = 0$ or the bracket $=0$ .

If a point is strictly inside its side ( $y_i(\mathbf w^\top\mathbf x_i+b) > 1$ ), the bracket $\neq 0$ , so we are forced to have $\alpha_i = 0$ → it contributes nothing to $\mathbf w$ .
Only points on the margin ( $y_i(\mathbf w^\top\mathbf x_i+b)=1$ ) can have $\alpha_i>0$ → these are the support vectors.

Recovering the bias $b$

WHY: stationarity gives $\mathbf w$ but not $b$ . Use any support vector $\mathbf x_s$ (where the margin equals exactly 1): $y_s(\mathbf w^\top\mathbf x_s + b) = 1 \;\Rightarrow\; b = y_s - \mathbf w^\top \mathbf x_s \quad(\text{using } y_s=\pm1,\ y_s^2=1).$ In practice average over all SVs for numerical stability.

Soft margin: three flavors of support vector

With slack $\xi_i$ and box constraint $0\le \alpha_i \le C$ :

Location	Condition	$\alpha_i$
Beyond margin (safe)	$y_i f(\mathbf x_i) > 1$	$\alpha_i = 0$ (not SV)
Exactly on margin	$y_i f(\mathbf x_i) = 1$	$0 < \alpha_i < C$
Inside margin / misclassified	$y_i f(\mathbf x_i) < 1$	$\alpha_i = C$

Worked examples

Common mistakes

Flashcards

What algebraic condition defines a support vector?

Its Lagrange multiplier

\alpha_i > 0

(equivalently it lies on/inside the margin).

Why do non-support vectors not affect the decision boundary?

KKT complementary slackness forces

\alpha_i=0

for points strictly beyond the margin, so they contribute nothing to

\mathbf w=\sum\alpha_i y_i\mathbf x_i

Write

\mathbf{w}

in terms of training data.

\mathbf w=\sum_i \alpha_i y_i \mathbf x_i

, nonzero only over support vectors.

In the soft margin, what does

\alpha_i = C

indicate?

The point is a margin violator or misclassified (inside the margin / wrong side).

How do you recover

b

from a support vector

\mathbf x_s

b = y_s - \mathbf w^\top \mathbf x_s

(since on the margin

y_s(\mathbf w^\top\mathbf x_s+b)=1

What does a very large fraction of support vectors suggest?

Overfitting / small margin / poor generalization; SV fraction bounds LOO error.

Kernelized decision rule?

f(\mathbf x)=\operatorname{sign}(\sum_{i\in SV}\alpha_i y_i K(\mathbf x_i,\mathbf x)+b)

Recall Feynman: explain to a 12-year-old

You're playing tug-of-war between two teams, and you draw a chalk line in the middle. Only the kids standing right at the front of each team, closest to the line, decide where the line goes. The kids way at the back don't pull on the line at all. Those front kids are the "support vectors" — and if all the back kids went home, the line wouldn't move even one inch.

Connections

Maximal margin classifier — where the $1/\|\mathbf w\|$ margin comes from
SVM dual formulation — the $\alpha_i$ live here
KKT conditions — complementary slackness drives the interpretation
Kernel trick — $\mathbf x_i^\top\mathbf x \to K$ keeps the SV sum
Soft margin & slack variables — the $\alpha_i=C$ case
Bias-variance tradeoff — SV count as a complexity signal

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, SVM ka core idea simple hai: do classes ke beech me ek "gali" (street) banao jo jitni possible ho utni chaudi ho. Is gali ka border kaun decide karta hai? Sirf woh points jo bilkul edge pe khade hain — unhe hum support vectors kehte hain. Baaki jo points apni-apni side me deep andar khade hain, woh boundary ko ratti bhar bhi affect nahi karte. Agar unhe delete kar do, model exactly same rahega. Yahi SVM ki khoobsurti hai — sparse solution.

Maths me yeh baat KKT complementary slackness se aati hai: har point ke liye ya to $\alpha_i = 0$ hota hai, ya woh point margin pe exactly baitha hota hai. Jo points margin ke andar (safe zone me) hain, unka $\alpha_i$ zabardasti $0$ ho jaata hai. Aur $\mathbf w = \sum \alpha_i y_i \mathbf x_i$ — matlab weight vector sirf un points ka weighted sum hai jinka $\alpha>0$ , yaani sirf support vectors ka. Isliye final decision function me bhi sirf support vectors dikhte hain.

Soft margin me thoda twist hai: agar point margin ke andar ghus jaaye ya galat side pe ho, to uska $\alpha_i = C$ (ceiling pe pahunch jaata hai). Yeh troublemaker points bhi support vectors hote hain, sirf boundary ke paas wale nahi. Ek common galti yeh hai ki log sochte hain "closest points = support vectors" — par actual definition hai $\alpha_i > 0$ .

Practical tip: agar tumhare model me support vectors ki sankhya bahut zyada hai (jaise 950 out of 1000), to samajh jao ki model overfit kar raha hai ya margin bahut chhota hai. Kam support vectors = better generalization aur faster prediction. Isliye SV count ek accha health-check hai apne SVM ka.

Test yourself — SVM, Naive Bayes & Probabilistic Models

Connections