Elastic Net regularization — Hinglish notes

2.2.15 · AI-ML › Linear & Logistic Regression

WHAT is it?

WHY combine them? (first-principles reasoning)

L1 sparsity kyun deta hai: L1 ball ke corners axes par hote hain. Jab loss contour pehli baar is ball ko touch karti hai, woh aksar corner par touch karti hai → kuch coordinates exactly $0$ ho jaate hain.

L1 akela correlated features par kyun fail hota hai: agar $x_{1} \approx x_{2}$ ho, toh loss $w_{1} + w_{2} = const$ direction mein almost flat hoti hai. L1 ki penalty $∣ w_{1} ∣ + ∣ w_{2} ∣$ bhi us direction mein constant hoti hai (jab tak signs match karein), isliye solution unique nahi hota — woh arbitrarily hil-dol jaata hai.

L2 add karne se yeh kyun theek hota hai: L2 penalty $w_{1}^{2} + w_{2}^{2}$ strictly convex hai aur (fixed sum ke liye) minimize hoti hai jab $w_{1} = w_{2}$ ho. Toh L2 term tie todta hai aur weight ko correlated features ke beech evenly spread karta hai. Yahi grouping effect hai.

HOW to derive the update (soft-thresholding)

Hum ek weight $w_{j}$ ke liye coordinate-descent update derive karte hain. Assume karo ki features standardized hain, yaani $\frac{1}{n} \sum_{i} x_{ij}^{2} = 1$ .

Step 1 — $w_{j}$ ko isolate karo. Baaki sab weights fix rakho. Partial residual define karo (target minus baaqi sab ka contribution): $r_{i} = y_{i} - \sum_{k \neq = j} w_{k} x_{ik} .$ Yeh step kyun? Coordinate descent ek waqt mein ek variable optimize karta hai; jo bhi $w_{j}$ se involve nahi hai woh constant hai.

Step 2 — 1-D objective likho. $f (w_{j}) = \frac{1}{2 n} \sum_{i} (r_{i} - w_{j} x_{ij})^{2} + λ α ∣ w_{j} ∣ + \frac{λ ( 1 - α )}{2} w_{j}^{2}$ Yeh step kyun? Yeh exactly $J$ hai jismein sirf $w_{j}$ free hai.

Step 3 — subgradient lo aur 0 par set karo. $ρ_{j} = \frac{1}{n} \sum_{i} x_{ij} r_{i}$ (feature $j$ ka partial residual ke saath correlation). $\frac{1}{n} \sum x_{ij}^{2} = 1$ use karte hue: $0 \in - ρ_{j} + w_{j} + λ (1 - α) w_{j} + λ α \partial ∣ w_{j} ∣$ Yeh step kyun? Quadratic parts cleanly differentiate hote hain; $∣ w_{j} ∣$ ka subgradient $w_{j} \neq = 0$ ke liye $sign (w_{j})$ hai aur $0$ par $[- 1, 1]$ hai.

Step 4 — solve karo, soft-threshold milta hai. Minimizer hai: $w_{j} = \frac{S ( ρ _{j} , λ α )}{1 + λ ( 1 - α )}, S (z, γ) = sign (z) max (∣ z ∣ - γ, 0)$ Yeh step kyun? $S$ soft-thresholding operator hai: L1 term $ρ_{j}$ ko zero ki taraf $λ α$ se shrink karta hai (aur maar deta hai agar $∣ ρ_{j} ∣ \leq λ α$ ho), jabki L2 term $1 + λ (1 - α)$ se divide karta hai, jo ek extra proportional shrink hai.

Worked examples

Common mistake Steel-manned pitfalls

"Elastic Net sirf Lasso hai ek tiny extra term ke saath — hamesha $α = 0.99$ use karo." Kyun sahi lagta hai: aap mostly sparsity chahte ho, toh $α$ ko 1 ke paas rakhte ho. Fix: correlated features ke saath, thoda sa bhi L2 (jaise $α = 0.5$ – $0.8$ ) selection ko dramatically stabilize karta hai. $α$ ko gut se nahi, cross-validation se tune karo.

" $λ$ aur $α$ ek hi kaam karte hain." Kyun sahi lagta hai: dono penalty knobs hain. Fix: $λ$ = kitni total regularization; $α$ = kaisi (L1 vs L2 mix). Yeh orthogonal hain — aap ek 2-D grid tune karte ho.

"Features standardize karna bhool gaye." Kyun sahi lagta hai: linear regression scale-equivariant hai, toh kya fark padta hai. Fix: penalties scale-invariant nahi hain. Ek feature kilometers mein vs metres mein alag penalize hogi. Elastic Net se pehle hamesha standardize karo.

Active recall

Recall Forecast-then-Verify: pehle predict karo, phir reveal karo

Q: Agar $α = 0$ ho, toh kaun sa method milta hai, aur kya woh zeros produce karta hai? A: Pure Ridge; yeh shrink karta hai lekin weights ko kabhi exactly zero nahi karta.

Q: Do perfectly correlated features — Elastic Net unke saath weight ka kya karta hai? A: L2 term ki wajah se weight ko unke beech (roughly) equally split karta hai (grouping effect).

Recall Feynman: ek 12-saal ke bachhe ko explain karo

Socho tum ek trip ke liye backpack pack kar rahe ho aur tumhe decide karna hai ki kaun si gadgets laani hain. Lasso ek strict dost hai: "EK flashlight lo, baaki do phenko" — chahe teeno equally achhi hon, aur agli baar woh alag wali randomly pick kar sakta hai. Ridge ek soft dost hai: "sab ki chhoti-chhoti versions laao, koi ghar nahi chhutega." Elastic Net samajhdaar dost hai: "jo cheez sacchi bekaar hai usse phenko (jaise Lasso), lekin teeno equally-achhi flashlights ke liye, teeno chhoti laao taaki fair aur stable rahe (jaise Ridge)." Equally-achhi chezon ke liye yeh fairness hi wajah hai ki yeh similar chezon se ghabrata nahi.

The 80/20 core

$J = MSE + λ (α ∥ w ∥_{1} + \frac{1 - α}{2} ∥ w ∥_{2}^{2})$ .
Update = soft-threshold (numerator, L1) over $1 + λ (1 - α)$ (denominator, L2).
L1 → sparsity; L2 → grouping/stability; Elastic Net → dono.
$λ$ (strength) aur $α$ (flavor) ko CV se tune karo; pehle standardize karo.

Connections

Ridge Regression — $α = 0$ special case (sirf L2).
Lasso Regression — $α = 1$ special case (sirf L1).
Soft-Thresholding Operator — woh proximal operator jo update ko power deta hai.
Coordinate Descent — woh algorithm jo ise solve karta hai.
Bias-Variance Tradeoff — regularization variance ko bias ke liye trade karta hai.
Feature Correlation & Multicollinearity — woh problem jo Elastic Net specifically sambhalta hai.
Cross-Validation — $λ, α$ kaise choose kiye jaate hain.

Elastic Net loss mein kaun si penalty add hoti hai?

Ek convex combination

λ (α ∥ w ∥_{1} + \frac{1 - α}{2} ∥ w ∥_{2}^{2})

— L1 aur L2 dono.

Elastic Net mein

α

kya control karta hai?

L1 aur L2 ke beech mixing ratio;

α = 1

Lasso hai,

α = 0

Ridge hai.

λ

kya control karta hai?

Regularization ki overall strength (L1/L2 flavor se independent).

Lasso correlated features par kyun struggle karta hai?

Iska objective correlated direction mein flat hota hai, isliye selection non-unique/unstable hoti hai.

L2 term correlated features ko kyun fix karta hai?

L2 strictly convex hai aur equal weights par minimize hoti hai, isliye weight evenly split hoti hai (grouping effect).

w_{j}

ke liye coordinate-descent update likho.

w_{j} = S (ρ_{j}, λ α) / (1 + λ (1 - α))

jahan

S

soft-thresholding hai.

Soft-thresholding operator

S (z, γ)

kya hai?

sign (z) max (∣ z ∣ - γ, 0)

— zero ki taraf shrink karta hai, zero set karta hai agar

∣ z ∣ \leq γ

ho.

Update ka kaun sa part sparsity deta hai vs stability?

Numerator (L1 soft-threshold) → sparsity; denominator

1 + λ (1 - α)

(L2) → stability/grouping.

Kya Elastic Net se pehle features standardize karne chahiye?

Haan — penalties scale-invariant nahi hain, isliye unstandardized scales bias karti hain ki kaun si features penalize hongi.

Agar

ρ_{j} = 0.3

λ = 1

α = 0.5

ho, toh

w_{j}

kya hai?

0

, kyunki

∣0.3∣ < λ α = 0.5

toh soft-threshold zero hai.