Regularization — L1 (lasso), L2 (ridge), dropout: Exercises

5.6.3 · D4 · Coding › Machine Learning (Aerospace Applications) › Regularization — L1 (lasso), L2 (ridge), dropout

Yeh page Regularization — L1, L2, dropout ke liye ek graded workout hai. Yahaan use kiye gaye har symbol ko parent note mein build kiya gaya hai; agar koi bhool jaaye, wapas jump karo. Har problem ko uska solution kholne se pehle solve karo — poora point yahi hai ki apne khud ke gaps pakdo.

Quick symbol refresher, taaki pehli line samajh aa sake:

Recall Woh chhe symbols jo tumhe chahiye

$w$ (ya $w_{i}$ ) ::: ek weight — ek tunable number jo model kisi feature se multiply karta hai.
$η$ (eta) ::: learning rate — gradient descent kitna bada step leta hai.
$λ$ (lambda) ::: regularization strength — hum bade weights ko kitna hard punish karte hain.
$\nabla L_{data}$ ::: data gradient — plain (unpenalized) loss ka slope.
$p$ ::: dropout probability — training ke dauran ek neuron ke switch off hone ka chance.
$m$ ::: dropout mask — ek coin-flip variable, $m \sim Bernoulli (1 - p)$ , jo $1 - p$ probability ke saath $1$ (neuron rakhna) aur $p$ probability ke saath $0$ (drop karna) hota hai.

Woh chaar update rules jin par neeche sab kuch tika hua hai. Har ek ke saath why padho — penalty ki shape hi rule hai:

Formula Woh rules jo tum baar baar use karoge (saath mein "why" bhi)

L2 (ridge) step: $w \leftarrow (1 - η λ) w - η \nabla L_{data}$ Multiplicative factor kyun? L2 penalty hai $\frac{λ}{2} w^{2}$ ; iska slope hai $λ w$ — $w$ ke proportional. Us slope ko descend karna $η λ w$ subtract karta hai, yaani har step mein weight ka ek fixed fraction $η λ$ hat jaata hai, aur $(1 - η λ) w$ bachta hai. Bowl-shaped penalty bade weights par zyada push karta hai aur chote par halka → smooth shrinkage. Stability note: yeh tabhi cleanly shrink karta hai jab $0 < η λ < 1$ ; $η λ = 1$ par factor $0$ ho jaata hai (weight ek hi step mein collapse) aur $η λ \geq 2$ par sign flip / diverge ho jaata hai — Exercise 5.2 dekho.

L1 (lasso) step: $w \leftarrow w - η (\nabla L_{data} + λ sign (w))$ Constant kyun, fraction kyun nahin? L1 penalty hai $λ ∣ w ∣$ ; iska slope hai $λ sign (w)$ — ek constant $\pm λ$ jiska size $w$ par depend nahin karta. Toh har step zero ki taraf same amount $η λ$ subtract karta hai. V-shaped penalty ki steepness har jagah same hoti hai, isliye chote weights bhi full-strength push feel karte hain → woh zero cross karke wahan reh jaate hain (sparsity).

L1 soft-threshold (pure penalty, no data gradient): $w^{new} = sign (w) max (0, ∣ w ∣ - η λ)$ Max kyun? Chote $∣ w ∣$ se constant $η λ$ subtract karna zero se aage overshoot karke sign flip kar dega — matlab penalty ne weight ko grow kar diya. $max (0, \cdot)$ zero par clamp karta hai taaki weight sirf shrink ho, kabhi reverse na ho.

Inverted dropout (train): $\tilde{h} = m \cdot h / (1 - p), m \sim Bernoulli (1 - p);$ test: $h^{test} = h$ $(1 - p)$ se divide kyun? Ek dropped neuron $0$ contribute karta hai, toh average par layer ka total signal uski true value ka fraction $(1 - p)$ tak shrink ho jaata — next layer training mein test ke comparison mein systematically weaker input dekhta. Kept neurons ko $(1 - p)$ se divide karna expectation restore karta hai: $E [\tilde{h}] = (1 - p) \cdot \frac{h}{1 - p} + p \cdot 0 = h$ . Kyunki training-time expectation already raw $h$ ke barabar hai, test time par kisi rescaling ki zaroorat hi nahin.

Common mistake Woh corner case jo sablog bhool jaate hain: bias ko regularize mat karo

Regularization weights $w_{i}$ ko penalize karta hai, lekin standard convention hai bias term $b$ ko (aur aksar BatchNorm/LayerNorm scale-shift parameters ko bhi) penalty se exclude karna. Kyun? Bias poori prediction ko upar ya neeche shift karta hai — yeh kisi input feature ko multiply nahin karta, isliye yeh woh over-flexible, noise-fitting behaviour cause nahin kar sakta jise regularization target karta hai. Use penalize karna model ke baseline ko bina kisi fayde ke zero ki taraf force karta hai, predictions ko bias karta hai. Fix: $\sum_{i} w_{i}^{2}$ (ya $\sum_{i} ∣ w_{i} ∣$ ) mein, $i$ ko sirf weights par range karo; $b$ ko untouched chodo. PyTorch jaise frameworks mein yeh sahi tarah se karne ke liye parameter groups split karne padte hain.

Level 1 — Recognition

Exercise 1.1 (L1 · Recognition)

Har phrase ke liye technique ka naam batao (L1 / L2 / dropout): (a) "weights ke squared magnitude ko penalize karta hai" (b) "training ke dauran randomly neurons ko off switch karta hai" (c) "bahut saare weights ko exactly zero par drive karta hai" (d) "weight decay bhi kehte hain ise"

Recall Solution

(a) L2 (ridge) — squared term $\sum w_{i}^{2}$ . (b) dropout — activations ki Bernoulli masking. (c) L1 (lasso) — diamond corners zeroed coordinates par land karte hain. (d) L2 (ridge) — $(1 - η λ)$ factor har step mein weights shrink karta hai, yaani decay.

Exercise 1.2 (L1 · Recognition)

Ek network test time par hai. Standard inverted dropout ke under, activations ke saath kya karte ho?

Recall Solution

Kuch nahin. Test time full network use karta hai bina mask aur bina scaling ke, kyunki $1/ (1 - p)$ scaling training ke dauran already apply ho chuki thi. Toh $h^{test} = h$ .

Level 2 — Application

Exercise 2.1 (L2 · Application)

$η = 0.1$ , $λ = 0.5$ , weight $w = 4$ , aur data gradient $\nabla L_{data} = 2$ diya gaya hai. Ek L2 update step compute karo.

Recall Solution

$w \leftarrow (1 - η λ) w - η \nabla L_{data}$ use karo.

Decay factor: $1 - η λ = 1 - (0.1) (0.5) = 0.95$ .
$w_{new} = 0.95 \times 4 - 0.1 \times 2 = 3.8 - 0.2 = 3.6$ . Weight dono se shrank: decay se ( $4 \to 3.8$ ) aur data gradient se ( $- 0.2$ ).

Exercise 2.2 (L2 · Application)

Same $η = 0.1$ , $λ = 0.5$ , lekin ab data gradient band karo ( $\nabla L_{data} = 0$ ). $w = 4$ se shuru karke, $∣ w ∣ < 1$ hone mein kitne steps lagte hain? (Yeh dikhata hai ki pure decay geometric hoti hai.)

Recall Solution

Bina data gradient ke, har step $0.95$ se multiply karta hai: $w_{t} = 4 (0.95)^{t}$ . Chahiye $4 (0.95)^{t} < 1 \Rightarrow (0.95)^{t} < 0.25 \Rightarrow t > \frac{ln 0.25}{ln 0.95} = \frac{- 1.386}{- 0.0513} \approx 27.03$ . Kyunki $t$ poore steps count karta hai, $27.03$ ki value matlab hai step 27 abhi enough nahin — humein upar round karna hoga next integer par: $t = ⌈ 27.03 ⌉ = 28$ steps. (Jab "ek threshold tak kitne discrete steps" solve karo toh hamesha ceiling lo — fractional step exist nahin karta.) Isliye L2 weight shrink karta hai lekin kabhi zero nahin karta — geometric decay zero approach karta hai bina pahunche. (Note: yeh tab hi hold karta hai jab $0 < η λ = 0.05 < 1$ ; agar $η λ \geq 1$ toh geometric picture break ho jaati hai — Exercise 5.2 dekho.)

Exercise 2.3 (L1 · Application)

$η = 0.1$ , $λ = 2$ diya, toh $η λ = 0.2$ . Har weight par soft-threshold rule apply karo (pure penalty step): $w = [0.15, - 0.30, 0.05, - 1.00]$ .

Figure — Regularization — L1 (lasso), L2 (ridge), dropout

Recall Solution

Rule: $w^{new} = sign (w) max (0, ∣ w ∣ - 0.2)$ .

$0.15$ : $∣ w ∣ = 0.15 < 0.2 \Rightarrow max (0, - 0.05) = 0 \Rightarrow 0$ .
$- 0.30$ : $∣ w ∣ = 0.30$ , $0.30 - 0.2 = 0.10$ , sign $- 1 \Rightarrow - 0.10$ .
$0.05$ : $0.05 < 0.2 \Rightarrow 0$ .
$- 1.00$ : $1.00 - 0.2 = 0.80$ , sign $- 1 \Rightarrow - 0.80$ . Result: $[0, - 0.10, 0, - 0.80]$ . Do weights exactly kill ho gaye, bacche hue same constant $0.2$ se shrank.

Figure s01 (described): horizontal axis incoming weight $w$ hai, vertical axis uski new value hai. Ek dashed black diagonal "do nothing" identity line hai ( $new = w$ ). Red curve soft-threshold output hai: yeh poore band $- 0.2 < w < 0.2$ par (shaded red — "dead zone") zero par stuck flat line hai, phir identity line ke parallel split hoti hai lekin zero ki taraf $0.2$ shifted. Red band ke andar land hone wala koi bhi weight exactly zero par snap ho jaata hai; bahar wala survive karta hai lekin $0.2$ closer to zero pull hota hai.

Level 3 — Analysis

Exercise 3.1 (L3 · Analysis)

Do weights $w = 0.30$ par equal start karte hain. Ek L2 se train hota hai ( $1 - η λ = 0.9$ per step), doosra L1 se (constant $η λ = 0.05$ per step, soft-threshold). Data gradient ignore karo. Exactly 6 steps ke baad, kaun sa chhota hai, aur kaun sa (agar koi) zero hai?

Recall Solution

L2: $w_{6} = 0.30 (0.9)^{6} = 0.30 \times 0.531441 = 0.1594$ — abhi bhi positive, kabhi exactly zero nahin. L1: har step $0.05$ subtract karo: 6 steps ke baad hum $0.30$ subtract karte, lekin pehle zero hit hota hai.

$0.30 \to 0.25 \to 0.20 \to 0.15 \to 0.10 \to 0.05 \to 0.00$ . Yeh $6$ subtractions hain, step 6 par exactly $0$ par land karte hain. Toh L1 chhota hai (=0); L2 $0.1594$ hai. Yahi poori story hai: L1 sparsify karta hai, L2 sirf shrink karta hai.

Figure s02 (described): horizontal axis training step $t$ hai (0 se 6), vertical axis weight value hai. Ek black curve (circles) L2 hai: yeh bend karta hai, har point pichle ka $0.9 \times$ , flatten hota jaata hai zero line approach karte karte — lekin kabhi pahunchta nahin. Ek red line (squares) L1 hai: ek perfectly straight downhill ramp jo har step $0.05$ drop karta hai, exactly $t = 6$ par zero axis par strike karta hai, jahaan ek red dot landing mark karta hai. Straight red ramp vs bending black curve ek picture mein poora L1-vs-L2 contrast hai.

Exercise 3.2 (L3 · Analysis)

Constraint-region picture use karke explain karo, kyun L1 coordinates zero karta hai lekin L2 nahin karta. Diamond vs circle reference karo.

Recall Solution

Regularization = data loss minimize karo subject to weights par ek budget. Budget region hai:

L2: $w_{1}^{2} + w_{2}^{2} \leq C$ — ek smooth circle (koi special points nahin).
L1: $∣ w_{1} ∣ + ∣ w_{2} ∣ \leq C$ — ek diamond jiske corners axes par baithe hain (ek coordinate $= 0$ ). Data-loss contours ellipses hain jo tab tak expand karte hain jab tak pehli baar budget region ko touch na kar lein. Ek smooth circle generic point par touch hota hai → dono weights nonzero. Ek diamond ka sharp corner se touch hone ki highest probability hai → ek weight exactly zero. Yahi corner-catching geometric sparsity hai.

Figure s03 (described): origin par centred do black budget shapes hain — ek solid circle (L2) aur ek dashed diamond jiske chaar corners exactly $w_{1}$ aur $w_{2}$ axes par baithe hain (L1). Off to the right faint black concentric ellipses hain, data-loss contours, apne centre se baahir grow karte hue. Jahan growing ellipse pehli baar diamond se milti hai wahan ek red dot $w_{1}$ -axis par bilkul baithe hai, yaani $w_{2} = 0$ — ek sparse solution. Corner baahir reach karke contour "pakad" leta hai; smooth circle off-axis point par graze hota jahan dono weights nonzero hote.

Level 4 — Synthesis

Exercise 4.1 (L4 · Synthesis)

Ek hidden layer activation $h = 3.0$ output karta hai. Tum inverted dropout $p = 0.4$ ke saath use karte ho. (a) Training ke dauran, jab neuron kept hai, kaun si value aage flow karta hai? (b) Verify karo ki expected training activation raw $h$ ke barabar hai. (c) Test time par kya aage flow karta hai?

Recall Solution

Keep probability $= 1 - p = 0.6$ . Inverted rule: kept value $= h / (1 - p)$ . (a) $3.0/0.6 = 5.0$ . (b) $E [\tilde{h}] = (1 - p) \cdot \frac{h}{1 - p} + p \cdot 0 = h = 3.0$ . ✓ Expectation preserved hai — training ke dauran scale up karne ka yahi poora reason hai. (c) Test time koi mask nahin, koi scaling nahin use karta: $h^{test} = 3.0$ .

Exercise 4.2 (L4 · Synthesis)

Tumhe 50 candidate features se aircraft drag predict karna hai, lekin tum jaante ho ki sirf kuch physically relevant hain aur tum chahte ho ki model tumhe bataye kaun se hain. Tum survivors par smooth, stable coefficients bhi chahte ho. Kaun sa regularizer(s) choose karoge, aur kyun? cross-validation ke saath kya tune karoge?

Recall Solution

L1 (ya Elastic Net = L1 + L2) use karo:

L1 automatic feature selection perform karta hai — irrelevant 45-ish sensors zero kar deta hai, feature selection free mein karta hai aur interpretable results deta hai.
Ek small L2 term add karna surviving coefficients stable rakhta hai jab features correlated hain (pure L1 arbitrarily do correlated sensors mein se ek pick karta hai). $λ$ (aur L1/L2 mix) ko cross-validation se tune karo: $λ$ sweep karo, woh value pick karo jo validation error minimize kare. Confirm karo ki overfit nahin ho rahe overfitting detection se (train-vs-validation gap).

Level 5 — Mastery

Exercise 5.1 (L5 · Mastery)

Ek fault-detection network overfit kar raha hai: train accuracy 99%, test 78%. Tum fully-connected layers par dropout $p = 0.5$ add karte ho aur train 92%, test 89% milta hai. (a) Generalization gap (train − test) kitne points se shrank? (b) Training accuracy giri. Explain karo yeh expected kyun hai, bug nahin. (c) Connections list se ek alternative regularizer aur ek complementary diagnostic ka naam lo.

Recall Solution

(a) Pehle: gap $= 99 - 78 = 21$ pts. Baad mein: gap $= 92 - 89 = 3$ pts. Shrinkage $= 21 - 3 = 18$ points. (b) Har training step ab ek random sub-network run karta hai, toh model exact examples memorize nahin kar sakta — training accuracy zaroor giregi. Hum thoda training fit trade karte hain ek bade test gain ke liye: regularization ka poora objective yahi hai. (c) Alternative regularizer: L2 weight decay same fully-connected layers par (ya ensemble methods — dropout khud $\sim 2^{n}$ sub-networks ka ek implicit ensemble hai, toh real ensemble banana generalization same direction push karta hai). Complementary diagnostic: overfitting detection — train-vs-validation accuracy curves plot karo aur gap dekho; isse cross-validation ke saath pair karo taaki dropout rate $p$ actually choose kar sako, $0.5$ guess karne ki jagah.

Exercise 5.2 (L5 · Mastery)

Plain gradient-descent update se, L2 "weight decay" form derive karo, aur precisely woh condition state karo $η λ$ par jiske liye decay stable ho (weights blow up ya oscillate na karein).

Recall Solution

Total loss se shuru karo $L_{total} = L_{data} + \frac{λ}{2} \sum_{i} w_{i}^{2}$ (sum sirf weights par — bias $b$ convention se exclude hai). $w_{i}$ ke w.r.t. iska gradient: $\frac{\partial L _{total}}{\partial w _{i}} = \frac{\partial L _{data}}{\partial w _{i}} + λ w_{i}$ (woh $\frac{1}{2}$ exponent ka $2$ cancel karta hai — isliye hum wahan rakhte hain). $w_{i} \leftarrow w_{i} - η \frac{\partial L _{total}}{\partial w _{i}}$ mein plug karo: $w_{i} \leftarrow w_{i} - η (\frac{\partial L _{data}}{\partial w _{i}} + λ w_{i}) = (1 - η λ) w_{i} - η \frac{\partial L _{data}}{\partial w _{i}} .$ Factor $(1 - η λ)$ weight decay hai. Pure-decay part ki stability $w_{t + 1} = (1 - η λ) w_{t}$ : ratio $r = 1 - η λ$ ke saath geometric.

Monotonically $0$ ki taraf decay karta hai jab $0 < 1 - η λ < 1 \Rightarrow 0 < η λ < 1$ .
Boundary $η λ = 1$ : $r = 0$ , toh weight ek hi step mein exactly zero par collapse ho jaata hai — usable lekin drastic (poora decay, old value ki koi memory nahin).
Agar $1 < η λ < 2$ toh $- 1 < r < 0$ : weight har step sign flip karta hai (shrink hote hue oscillate); agar $η λ \geq 2$ toh $∣ r ∣ \geq 1$ aur yeh diverge karta hai. Toh clean shrinkage ke liye $η λ < 1$ rakho; $η λ = 1$ ko knife-edge treat karo aur usse upar kuch bhi unsafe maano.

Recall Self-check summary

L1 vs L2 ek line mein? ::: L1 ek constant subtract karta hai (→ sparsity), L2 $(1 - η λ)$ se multiply karta hai (→ smooth shrinkage, kabhi zero nahin). $sign (0)$ kya hai aur kyun matter karta hai? ::: Formally undefined (subgradient $[- 1, 1]$ mein kahin bhi); optimizers ise $0$ set karte hain taaki zero par pahuncha weight wahan ruk jaaye — isliye soft-thresholding safe form hai. Test time par dropout? ::: Kuch nahin — inverted dropout ne training ke dauran already scale kar diya. Training ke dauran $(1 - p)$ se divide kyun karte hain? ::: Expected activation preserve karne ke liye ( $E [\tilde{h}] = h$ ) taaki test par koi rescaling na chahiye. L2 stability range? ::: Clean decay ke liye $0 < η λ < 1$ ; $η λ = 1$ weight ko ek step mein zero par collapse karta hai; $η λ \geq 2$ diverge karta hai. Kya hum bias regularize karte hain? ::: Nahin — convention se $b$ penalty se exclude hai kyunki yeh shift karta hai, scale nahin, aur overfitting cause nahin kar sakta.