A deep net has millions of parameters. With that much freedom it can memorize noise in the training set (huge, spiky weights that carve out weird bumps to hit every training point). That kills generalization.
The cure: add a penalty on weight size to the loss. Now the optimizer must trade off:
lowering the data loss (fit the points), vs.
keeping ∥w∥ small (stay simple).
This is exactly the bias–variance tradeoff: we accept a little more bias to cut variance.
Start from plain gradient descent on the regularized loss with L2:
J~(w)=J(w)+2λ∥w∥22
Take the gradient:
∇wJ~=∇wJ+λwWhy this step?∂wi∂(2λwi2)=λwi, so the whole penalty gradient is just λw.
Now the update rule with learning rate η:
w←w−η(∇wJ+λw)
Rearrange:
w←shrink factor(1−ηλ)w−η∇wJ
Why this is beautiful: even before any data pushes the weights, they leak toward zero each step. A weight survives only if the data gradient keeps re-supplying it.
Look at the L1 gradient:
∇w(λ∥w∥1)=λsign(w)Why this step?dwd∣w∣=sign(w) (undefined at 0, handled by subgradient).
The update becomes:
w←w−η∇wJ−ηλsign(w)
Notice the penalty term ηλsign(w) is a constant push toward zero of fixed size, regardless of how big w is. For L2, the push ηλwshrinks as w→0, so it never quite reaches zero. For L1 the push stays full-strength until it crosses zero — so many weights get snapped exactly to 0.
The classic picture: minimizing data loss (elliptical contours) subject to a budget on weights. The L1 budget is a diamond (corners on the axes), the L2 budget is a circle. The loss contour first touches the diamond at a corner (a weight = 0), but touches the circle anywhere.
Imagine you're packing a backpack for a hike. Every item is a "weight" the network wants to carry. Weight decay is a rule: "You can bring anything, but each item costs you energy, so only pack it if it truly helps."L2 says every item slowly gets lighter each step, so useless items fade away but nothing is fully banned. L1 says every item costs a fixed toll, so the almost-useless ones get thrown out completely — you end up with a small, tidy backpack. A tidy backpack (simple network) is easier to carry on a new trail (new data).
Dekho, deep network mein millions of weights hote hain, aur itni freedom ke saath network training data ko ratta maar leta hai — noise tak fit kar deta hai, spiky weird function ban jaata hai. Yeh overfitting hai. Iska solution: loss ke saath weights ke size ka ek penalty jod do. Ab optimizer ko do cheezein balance karni padti hain — data fit karo, par weights ko chhota rakho. Isko regularization ya weight decay kehte hain, aur strength ko λ control karta hai.
L2 (2λ∥w∥2) ka magic yeh hai: update likho to ban jaata hai w←(1−ηλ)w−η∇J. Yaani har step mein weight pehle ek factor (1−ηλ) se chhota ho jaata hai — isliye naam "weight decay"! Weight tabhi zinda rehta hai jab data usko baar-baar wapas push kare. Iska equilibrium point hai w⋆=−∇J/λ — matlab data ko har weight "kamaana" padta hai.
L1 (λ∥w∥1) alag kaam karta hai. Iska gradient hai λsign(w) — ek constant force jo weight ke size par depend nahi karti. Isliye chhote weights ko yeh seedha zero par patak deti hai. Result: sparse network (bahut saare weights exactly 0). L2 sab weights ko chhota karta hai par zero nahi; L1 kuch ko poori tarah kaat deta hai — feature selection jaisa.
Ek galti se bacho: λ bahut bada mat karo, warna sab weights zero ki taraf bhaag jaayenge aur underfitting ho jaayega. Aur yaad rakho — Adam ke saath simple L2 same nahi hota decay ke; isliye AdamW use karte hain jo decay ko alag rakhta hai. Validation set par λ tune karo, bas.
