Regularization — L1 (lasso), L2 (ridge), dropout: Visual walkthrough

5.6.3 · D2 · Coding › Machine Learning (Aerospace Applications) › Regularization — L1 (lasso), L2 (ridge), dropout

Yeh page regularization ki sabse important picture ko bilkul scratch se rebuild karta hai: diamond-shaped penalty (L1) weights ko exactly zero par snap kyun karta hai, jabki round penalty (L2) unhe sirf chhota karta hai? Hum maante hain ki aapne pehle kabhi contour, norm, ya gradient nahi dekha. Hum ek ek cheez, ek ek karke, picture ke upar banate hain.

Step 1 — Do knobs, aur "the loss" asal mein kya hai

KYA HAI. Hamare model mein sirf do adjustable numbers hain, weights $w_{1}$ aur $w_{2}$ . Inhe do dials ki tarah socho. Har possible pair $(w_{1}, w_{2})$ kaagaz ki ek flat sheet par ek single point hai — horizontal axis $w_{1}$ hai, vertical axis $w_{2}$ hai.

KYUN. Penalties ya shapes ki baat karne se pehle, hume unhe draw karne ke liye ek stage chahiye. Woh stage saare possible weight choices ka plane hai. Hum do weights use karte hain (ek nahi, pachaas nahi) sirf isliye kyunki ek plane sabse badi cheez hai jo hum honestly draw kar sakte hain.

PICTURE. Figure dekhiye: sheet choices ka space hai. Ek dot mark hai — woh ek particular model hai, maano $(w_{1}, w_{2}) = (2, 1.5)$ .

Figure — Regularization — L1 (lasso), L2 (ridge), dropout

ek point (w_{1}, w_{2}) = dials ki ek fully-decided pair ek model

Step 2 — Data loss ek bowl hai; uska shadow ellipses ka ek nest hai

KYA HAI. Har choice $(w_{1}, w_{2})$ ke liye ek number hota hai: model training data ko kitna bura fit karta hai. Use $L_{data}$ bolein. Chhota = achha fit. Agar hum woh number sheet se upar plot karein, toh hume ek bowl milta hai — ek jagah low (best fit) aur door jaane par upar uthta hua.

KYUN. Parent note ne data loss ko per-example errors ka average likha tha,

L_{data} = \frac{1}{N} n = 1 \sum N ℓ (y_{n}, \overset{y}{^}_{n}) .

Hum yahan, is page par, har symbol define karte hain:

$y_{n}$ ::: training example $n$ ka sach wala target (jaise measured drag).
$\overset{y}{^}_{n}$ ::: us example ke liye model ki prediction — hamare linear model ke liye $\overset{y}{^}_{n} = w_{1} x_{1, n} + w_{2} x_{2, n}$ , toh yeh dials par depend karta hai.
$ℓ (y_{n}, \overset{y}{^}_{n})$ ::: per-example loss, ek number jo batata hai ek prediction kitna miss hua; hum squared miss use karte hain $ℓ = (y_{n} - \overset{y}{^}_{n})^{2}$ .
$\frac{1}{N} \sum_{n = 1}^{N}$ ::: saare $N$ training examples par us miss ka average.

Kyunki $\overset{y}{^}_{n}$ $w_{1}, w_{2}$ mein linear hai, ek squared miss dials mein quadratic hai, aur do variables mein ek quadratic hamesha ek bowl (paraboloid) hota hai. Hume yeh shape isliye chahiye kyunki yeh hume woh ek tool deta hai jo picture ko decode karta hai: contour lines.

PICTURE. Hum bowl ko equal heights par slice karte hain aur slices ko sheet par flat drop karte hain. Har slice ek ellipse hai — "equally-bad" models ki ek ring. Tiny centre dot $\overset{w}{^}$ unconstrained best fit hai (bowl ka bottom).

L_{data} (w_{1}, w_{2}) = ek fixed number (ek chosen height) c ⟹ saare models jo badness c par tied hain ek ellipse

Yahan $c$ sirf ek constant level value hai jo hum choose karte hain — "mujhe har woh dial-setting dikhao jiska badness exactly $c$ ke barabar hai." $c$ ko upar sweep karo aur tumhe rings ka poora nest milta hai.

Ellipse kyun hoti hai circle nahi? Kyunki data aksar dials ke ek combination ko doosre se zyada tightly pin karta hai, toh bowl weight space mein ek direction ke along steeper hota hai aur doosre ke along flatter. Steeper direction matlab loss fast barhti hai (ek squashed ring); flatter direction dheere barhti hai (ek stretched ring). Ek lopsided bowl lopsided rings deta hai — yahi "ellipse" ka matlab hai yahan.

Step 3 — "Bade weights pasand nahi" ke liye ek ruler chahiye: the norm

KYA HAI. Ab hum ek doosri wish add karte hain: weights ko chhota rakho. Kisi cheez ko chhota rakhne ke liye pehle uski size measure karni hoti hai. Ek rule jo ek pair $(w_{1}, w_{2})$ ko ek single "size" number mein badalta hai use norm kehte hain.

KYUN. Bigness ko penalize nahi kar sakte jab tak bigness define na ho — aur use measure karne ke ek se zyada honest tarike hain. Ruler ka yahi choice L1 aur L2 ko alag karta hai. Toh hum do rulers ek saath introduce karte hain.

PICTURE. Do rulers, "size 1 ke saare points" ki do shapes:

L2 ruler (origin se seedhi-line distance): $∥ w ∥_{2} = w_{1}^{2} + w_{2}^{2}$ . $1$ distance par saare points ek circle banate hain.
L1 ruler (sideways + upar moves add karo, jaise city blocks mein chalna): $∥ w ∥_{1} = ∣ w_{1} ∣ + ∣ w_{2} ∣$ . Size $1$ ke saare points ek diamond banate hain.

Term-by-term: L2 ruler Pythagoras hai (ek seedha flight). L1 ruler diagonals forbid karta hai — tumhe axes ke along move karna padta hai, toh ek diagonal zyada cost karta hai. Diagonals par yeh extra cost precisely wahi hai jo diamond ke corners ko axes par push karta hai.

Step 4 — Constraint view: origin ke around ek fence

KYA HAI. "Weights chhote rakho" kehne ke do equivalent tarike hain. Ek: loss mein $λ \cdot ∥ w ∥$ ki penalty add karo (parent note ka view, jahan $λ$ = regularization strength). Equivalent, aur aasaan-to-draw tarika: weights ko origin ke around size $C$ ki ek fenced region se bahar jaane se rokna. Formally, $L_{data}$ minimise karo subject to $∥ w ∥ \leq C$ .

KYUN. Penalty view page se upar exist karta hai (bowl ki height mein ek bump add karta hai). Fence view page par flat exist karta hai (ek region jise tum chhhod nahi sakte). Flat drawable hai, toh hum use choose karte hain.

Do views same trade-off kyun hain (intuitive sketch). Maano constrained answer fence par land karta hai (hota hai, jab bhi $\overset{w}{^}$ bahar ho). Us touching point par dono views ko "better fit" aur "chhota rehna" ke beech balance par agree karna chahiye. Fence ko thoda bahar push karo ( $C$ badhao): tumhe thoda bade weights allowed hain, toh thoda better fit kar sakte ho — exactly wahi jo penalty $λ$ ghataana karne deta hai. Fence shrink karo ( $C$ ghato): tumhe chhote weights ke liye bura fit accept karna hoga — exactly wahi jo $λ$ badhana force karta hai. Toh ek monotone pairing hai:

PICTURE. Har ruler ke liye fence draw karo:

L2 fence = radius $C$ ka ek filled disc.
L1 fence = ek filled diamond jiske corners $(\pm C, 0)$ aur $(0, \pm C)$ par hain.

Best fit $\overset{w}{^}$ dono fences ke bahar baithi hai (isliye hume regularization ki zaroorat thi). Toh hum $\overset{w}{^}$ par nahi baith sakte; hume fence ko touch karte hue best allowed point dhundhna hai.

w min L_{data} (w) subject to fence ke andar rehna ∥ w ∥ \leq C

$min_{w}$ ::: saare dial-settings par search karo.
$∥ w ∥ \leq C$ ::: allowed region — sirf fence ke andar ya uske upar.

Step 5 — Jahan ek ellipse pehli baar fence ko kiss karti hai

KYA HAI. Ellipses ko $\overset{w}{^}$ par tiny shuru karo aur unhe bahar grow hone do. Pehli jagah jahan ek growing ellipse fence ko touch karti hai woh hamaara answer hai: woh lowest-loss point hai jahan hum abhi bhi jaane allowed hain.

KYUN. Ellipse grow karna = zyada data-error accept karna. Hum kam se kam extra error chahte hain, toh jaise hi fence reach karo rukh jao. Jo point ellipse pehle touch kare woh constrained solution hai — koi picture-free algebra nahi chahiye.

PICTURE (L2, circle). Ellipse bahar roll karti hai aur round fence ko uski smooth edge par ek point par pehli baar graze karti hai. Wahan kuch special nahi hota: $w_{1}$ aur $w_{2}$ dono generally nonzero hain, sirf $\overset{w}{^}$ se chhote. Round fence → smooth touch → shrunk, but nonzero.

Step 6 — Diamond corner par kiss karta hai → exact zero

KYA HAI. Step 5 ko diamond fence ke saath repeat karo. Ab growing ellipse almost hamesha diamond ke sharp corners mein se ek par milti hai — aur L1 diamond ka ek corner exactly ek axis par baithi hai, jahan ek weight precisely $0$ hai.

KYUN. Ek corner incoming ellipses ki taraf sabse aage nikla hota hai. Bahar se sweep karti ek ellipse us bahar nikle corner tak pahunchu flat sides tak pahunchne se pehle. Yeh pointy shapes ke baare mein ek geometric fact hai, luck nahi. Aur $∣ w_{1} ∣ + ∣ w_{2} ∣ \leq C$ ke corners char points hain $(\pm C, 0), (0, \pm C)$ — har ek mein ek coordinate zero ke barabar hai.

PICTURE. Ellipse top corner $(0, C)$ ko touch karti hai: wahan $w_{1} = 0$ exactly. Model dial 1 ko completely drop kar deta hai — automatic feature selection, live, ek picture mein.

Yahi corner-snapping algebraically ek soft-threshold rule ke roop mein dikhai deta hai jab hum gradient descent se optimise karte hain. Zero dekhne ke liye hume uski zaroorat nahi — picture ne de diya — lekin iska naam batana worth hai, aur ise ek naye symbol ki zaroorat hai:

Yeh rule hai, parent note ke L1 section mein derive kiya gaya (hum sirf quote aur read karte hain):

L2 rule $w_{i}^{new} = (1 - η λ) w_{i}$ se compare karo: woh $1$ se thoda kam factor se multiply karta hai, toh — bar bar half karne ki tarah — woh zero ki taraf scale karta hai lekin kabhi wahan land nahi karta. Round fence, round rule, no exact zeros.

Step 7 — Degenerate & edge cases (kuch bhi chhupa nahi)

Hume un corners ko cover karna hai jo reader encounter kar sakta hai. Har ek ko figure mein apna panel milta hai.

Case A — $λ = 0$ (koi fence nahi). Fence ka infinite size hai ( $C = ∥ \overset{w}{^} ∥$ kaafi hai); ellipse $\overset{w}{^}$ par khushi se baithe hai. Koi shrinkage nahi. Overfitting risk wapas aa jaata hai.

Case B — $λ \to \infty$ (fence ek dot tak shrink ho jaata hai). $C \to 0$ : sirf ek allowed point origin hai. Dono weights zero hain — model ek constant predict karta hai. Useless, jaisa parent note ne warn kiya.

Case C — ellipse L1 edge ko touch karti hai, corner ko nahi. Yeh ho sakta hai: agar loss ellipses bilkul sahi aligned hon, toh pehla touch ek flat side par hota hai, do nonzero weights deta hai. Toh L1 zeros guarantee nahi karta — yeh unhe bahut zyada likely banata hai, kyunki corners almost har direction se pehle reach hote hain.

Case D — $w_{i} = 0$ exactly, jahan $∣ w_{i} ∣$ mein ek kink hai. Absolute-value ruler ka $0$ par koi single slope nahi hai (ek sharp "V"). Hum ise subgradient se handle karte hain: wahan $- 1$ aur $+ 1$ ke beech koi bhi slope allowed hai. Woh undefined slope exactly wahi hai jo ek weight ko zero par rest karne deta hai instead of off push hone ke — yeh ek bug nahi, feature hai.

Ek-picture summary

Final figure dono fences ko same incoming ellipses ke neeche side by side rakhti hai:

Left (L2, circle): smooth touch, dono weights shrunk, koi zero nahi → weight decay.
Right (L1, diamond): axis par corner touch, ek weight exactly zero → sparse selection.

Same loss, same $λ$ -sized fence — sirf ruler ki shape badi, aur us shape ne sab kuch decide kar diya.

Recall Feynman retelling — 12-saal ke bacche ko bolte huye

Tumhare paas do dials hain aur tum chahte ho machine flight fuel guess kare. Kuch dial settings achha guess karti hain, kuch bura; use kaagaz par rings ki tarah draw karo — inner rings achhe guesses hain, outer rings bure. Best guess ek dot hai, lekin woh dot huge, twitchy dials use karta hai jo nayi flights par toot jaate hain. Toh hum safe middle ke around ek fence draw karte hain aur kehte hain "tumhe andar rehna hai." Ab good-guess rings ko grow hone do jab tak koi fence ko touch na kare — woh touch tumhara answer hai. Agar fence circle hai, ring smoothly touch karti hai aur dono dials chhote-par-on aate hain. Agar fence diamond hai, ring almost hamesha ek pointy corner ko hit karti hai, aur woh corners seedha un lines par baithe hain jahan ek dial completely OFF hai. Isliye diamond ruler (L1) useless dials throw karta hai, aur round ruler (L2) sirf sab ko ek notch down karta hai. Fence ko tiny karo aur dono dials mar jaate hain; fence hatao aur tum wapas twitchy overfitting par ho. Fence size choose karna hi sirf real decision hai.

Recall

Round (L2) fence weights ke saath kya karta hai? ::: Dono ko smoothly zero ki taraf shrink karta hai, lekin rarely exactly zero tak (weight decay). Diamond (L1) fence exact zeros kyun produce karta hai? ::: Uske corners axes par lie karte hain, aur ek growing loss-ellipse almost har direction se pehle ek corner ko touch karti hai, ek weight ko exactly 0 force karti hai. In pictures mein $\overset{w}{^}$ kya hai? ::: Unconstrained best fit — data-loss bowl ka bottom, ellipses ka center. Fence picture mein bada $λ$ kya matlab hai? ::: Chhota fence $C$ — ek tighter constraint jo weights ko zyada shrink karta hai. Soft-threshold rule mein $η$ kya hai? ::: Learning rate — woh step size jo gradient descent har update mein leta hai. Ek aisa case batao jahan L1 kuch bhi zero nahi karta. ::: Jab ellipses diamond ki flat edge ko touch karti hain corner ki jagah, dono weights nonzero rehte hain (Case C).

Prerequisite & neighbour links: Gradient Descent Variants (update rules jinka geometry humne draw kiya), Feature Engineering (L1 ka automatic selection), Overfitting Detection (fence kyun lagaate hain), Ensemble Methods aur Neural Network Architectures (dropout as sub-networks ka ensemble), Bayesian Inference (penalties as priors), aur $λ$ ko Cross-Validation se choose karo.