L1 - L2 weight decay in deep nets — Hinglish notes

3.2.12 · AI-ML › Training Deep Networks

WHY: hum weights ko regularize kyun karte hain?

Ek deep net mein millions of parameters hote hain. Itni freedom ke saath woh training set ke noise ko memorize kar sakta hai (bahut bade, spiky weights jo har training point ko hit karne ke liye weird bumps banate hain). Isse generalization khatam ho jaati hai.

Ilaaj: loss mein weight size ka penalty add karo. Ab optimizer ko trade-off karna padega:

data loss kam karna (points ko fit karna), vs.
$∥ w ∥$ ko chota rakhna (simple rehna).

Yahi bias–variance tradeoff hai: hum thodi zyada bias accept karte hain taaki variance cut ho sake.

WHAT: L1 aur L2 penalties kya hain?

HOW: L2 literally "weight decay" kaise banta hai — derive karte hain

L2 ke saath regularized loss par plain gradient descent se shuru karte hain:

$\tilde{J} (w) = J (w) + \frac{λ}{2} ∥ w ∥_{2}^{2}$

Gradient lete hain: $\nabla_{w} \tilde{J} = \nabla_{w} J + λ w$ Yeh step kyun? $\frac{\partial}{\partial w _{i}} (\frac{λ}{2} w_{i}^{2}) = λ w_{i}$ , isliye poore penalty ka gradient bas $λ w$ hai.

Ab learning rate $η$ ke saath update rule: $w \leftarrow w - η (\nabla_{w} J + λ w)$

Rearrange karte hain: $w \leftarrow shrink factor (1 - η λ) w - η \nabla_{w} J$

Yeh kyun beautiful hai: data ke koi bhi weight push karne se pehle, weights har step mein zero ki taraf leak karte hain. Ek weight tabhi survive karta hai jab data gradient usse baar baar re-supply karta rahe.

HOW: L1 sparsity kaise produce karta hai — geometry

L1 gradient dekho: $\nabla_{w} (λ ∥ w ∥_{1}) = λ sign (w)$ Yeh step kyun? $\frac{d}{d w} ∣ w ∣ = sign (w)$ (0 par undefined, subgradient se handle hota hai).

Update ban jaata hai: $w \leftarrow w - η \nabla_{w} J - η λ sign (w)$

Dhyan do ki penalty term $η λ sign (w)$ zero ki taraf constant push hai, chahe $w$ kitna bhi bada ho. L2 mein, push $η λ w$ tab shrink karta hai jab $w \to 0$ , isliye woh kabhi exactly zero nahi pahunchta. L1 mein push full-strength rehta hai jab tak zero cross na ho jaaye — isliye bahut saare weights exactly 0 par snap ho jaate hain.

Classic picture: data loss (elliptical contours) minimize karo, weights par budget ke constraint ke saath. L1 budget ek diamond hai (corners axes par), L2 budget ek circle hai. Loss contour pehle diamond ko uske corner par touch karta hai (ek weight = 0), lekin circle ko kahin bhi touch kar sakta hai.

Figure — L1 - L2 weight decay in deep nets

Bayesian WHY (choice ko justify karna)

Penalty add karna = weights par prior rakhna aur MAP estimation karna:

L2 ⇔ Gaussian prior $w \sim N (0, 1/ λ)$ (log-prior $\propto - λ w^{2}$ ).
L1 ⇔ Laplace prior (0 par peaked, heavy tails) → exact zeros encourage karta hai.

Isliye regularization koi hack nahi hai — yeh ek belief statement hai: "zyaatar weights near zero hone chahiye."

Worked examples

Common mistakes

The 80/20 core

Recall Feynman: 12-saal ke bachche ko explain karo

Socho tum hiking ke liye backpack pack kar rahe ho. Har cheez ek "weight" hai jo network carry karna chahta hai. Weight decay ek rule hai: "Kuch bhi la sakte ho, lekin har cheez mein tumhari energy lagti hai, isliye tabhi pack karo jab sach mein help kare." L2 kehta hai har cheez har step thodi halki ho jaati hai, isliye useless cheezein fade ho jaati hain lekin kuch fully ban nahi hota. L1 kehta hai har cheez par fixed toll lagta hai, isliye almost-useless cheezein completely bahar ho jaati hain — tumhara backpack chhota aur tidy ho jaata hai. Tidy backpack (simple network) ko nayi trail (naya data) par carry karna aasaan hai.

Flashcards

L2 loss mein kaunsa penalty term add hota hai?

\frac{λ}{2} ∥ w ∥_{2}^{2}

L2 gradient-descent update ko "decay" form mein derive karo.

w \leftarrow (1 - η λ) w - η \nabla_{w} J

;

(1 - η λ)

factor har step weights ko shrink karta hai.

L2 ko literally "weight decay" kyun kehte hain?

Kyunki har update pehle

w

< 1

factor se multiply karta hai, isliye weights zero ki taraf decay karte hain jab tak data unhe re-supply na kare.

L1/L2 mein se kaunsa exact zeros produce karta hai aur kyun?

L1; iska gradient

λ sign (w)

hai — constant-magnitude push jo chote weights ko bilkul 0 tak drive karta hai.

L2 exact zeros kyun produce nahi karta?

Iska push

λ w

proportionally

w

ke saath shrink karta hai,

w \to 0

par vanish ho jaata hai, isliye weights tiny ho jaate hain lekin exactly zero nahi.

L2 ke under equilibrium weight (zero data gradient case)?

w^{⋆} = - \frac{1}{λ} \nabla_{w} J

; bada

λ

⇒ chota weight.

L2 ke saath kaunsa prior correspond karta hai? L1 ke saath?

L2 ⇔ Gaussian prior; L1 ⇔ Laplace prior.

Bias terms ko regularize kyun nahi karte?

Yeh sirf function ko shift karte hain; unhe penalize karna variance reduce kiye bina bias add karta hai aur fit kharab kar sakta hai.

Agar

λ

bahut bada ho toh kya hota hai?

Saare weights 0 ki taraf shrink ho jaate hain ⇒ underfitting (high bias).

AdamW, Adam + L2 se alag kyun hai?

AdamW weight decay ko adaptive gradient scaling se decouple karta hai, isliye decay per-parameter learning rates se distort nahi hoti.

L1 ke liye soft-threshold operator kya hai?

w \leftarrow sign (w) max (∣ w ∣ - η λ, 0)

Connections

Bias-Variance Tradeoff — regularization bias ko variance se trade karta hai.
Overfitting and Generalization — woh problem jise weight decay attack karta hai.
Maximum a Posteriori (MAP) Estimation — penalties as priors.
Adam and AdamW Optimizers — decoupled weight decay.
Dropout — ek alternative regularizer.
Gradient Descent — woh update jise humne modify kiya.
Lasso and Ridge Regression — same math linear case mein.