Worked examples — Regularization — L1 (lasso), L2 (ridge), dropout

5.6.3 · D3 · Coding › Machine Learning (Aerospace Applications) › Regularization — L1 (lasso), L2 (ridge), dropout

Yeh page parent topic ki practice arena hai. Humne teen tools — L2 (ridge), L1 (lasso), dropout — pehle se first principles se build kar liye hain. Ab hum inhe haath se har us situation mein chalate hain jo aa sakti hai: har sign, tricky exactly-at-zero case, dono limits $\lambda =0$ aur $\lambda \to \infty$, ek real aerospace word problem, aur ek exam-style twist.

Kuch bhi naya assume nahi kiya gaya. Agar koi symbol aata hai, toh woh parent note mein define tha; lekin hum usse wahan phir se state karte hain jab bhi use karte hain, toh tum line one se padh sakte ho.

Do symbols jinhe hum baar baar use karte hain:

• $\eta$ (eta) = learning rate, ek gradient step ka size. Socho "kitna bada kadam."
• $\lambda$ (lambda) = regularization strength, hum bade weights ko kitna penalize karte hain. Socho "teacher kitna strict hai."

---

The scenario matrix

Neeche har cell ek tarah ki situation hai jo topic throw kar sakta hai. Har worked example un cell(s) ke saath tagged hai jo woh cover karta hai, toh end tak har cell filled hogi.

#	Cell (case class)	Tool	Covered by
A	Positive weight, gentle shrink	L2	Ex 1
B	Negative weight, sign handled	L2	Ex 1
C	Limit $\lambda =0$ (no reg)	L2/L1	Ex 2
D	Limit $\lambda \to \infty$ (over-shrink / overshoot)	L2	Ex 2
E	Small weight → crosses zero (soft threshold)	L1	Ex 3
F	Large weight → shrinks but survives, both signs	L1	Ex 3
G	Degenerate input $w=0$ exactly (subgradient)	L1	Ex 4
H	Geometry: why L1 corner = sparsity	L1 vs L2	Ex 5
I	Dropout expectation / scaling both conventions	Dropout	Ex 6
J	Real-world word problem (choose the tool)	all	Ex 7
K	Exam twist (L1+L2 = elastic net, hidden trap)	mixed	Ex 8

---

Example 1 — L2 ek positive aur ek negative weight par (cells A, B)

Forecast: compute karne se pehle guess karo — positive weight chhota hona chahiye (0 ki taraf), negative weight kam negative hona chahiye (woh bhi 0 ki taraf). Regularization hamesha zero ki taraf pull karti hai. Yeh prediction hold karo.

Recipe (parent ki L2 formula se): $w^{\text{new}}=(1-\eta \lambda )w-\eta \frac{\partial {\mathcal{L}}_{\text{data}}}{\partial w}$

Step 1 — shrink factor $(1-\eta \lambda )$ compute karo. $1-\eta \lambda =1-(0.1)(0.5)=0.95$ Yeh step kyun? Yeh akela number L2 ki poori rooh hai: data dekhne se pehle, har weight apne aap ka $95\%$ ho jaata hai. Yahi "weight decay" hai.

Step 2 — isse $w_a=+4.0$ par apply karo. $0.95\times 4.0=3.8,3.8-(0.1)(1.0)=3.7$ Toh $w_a^{\text{new}}=3.7$. Yeh step kyun? Shrink ($4.0\to 3.8$) proportional hai — bada weight bada chunk kho deta hai. Phir data gradient use thoda aur neeche nudge karta hai.

Step 3 — isse $w_b=-2.0$ par apply karo. $0.95\times (-2.0)=-1.9,-1.9-(0.1)(-0.5)=-1.9+0.05=-1.85$ Toh $w_b^{\text{new}}=-1.85$. Yeh step kyun? Dekho sign automatically handle ho gaya — shrink factor jo bhi sign weight ka ho use multiply karta hai, toh $-2.0$ ban jaata hai $-1.9$ (zero ke karib, door nahi). L2 mein negatives ke liye koi special case ki zarurat nahi.

Verify: Dono zero ki taraf move hue: $|4.0|\to 3.7$ aur $|-2.0|\to 1.85$. Forecast se match. Bade weight ne $0.3$ decay kiya, chhote ne $\approx 0.15$ — proportional, jaisa promise tha. Units: weights dimensionless model coefficients hain, toh sab terms dimensionless hain. ✓

---

Example 2 — dono L2 limits (cells C, D)

Forecast: $\lambda =0$ ke saath kuch nahi hilna chahiye. Jaise $\lambda$ badhta hai weight zero ki taraf kheecha jaata hai. Lekin kya koi $\lambda$ itna bada hai jo overshoot kare? Guess: haan — aur socho kaun sa sign dikhayega.

Kyunki data gradient $0$ hai, update purely $w^{\text{new}}=(1-\eta \lambda )w$ hai.

Step 1 — $\lambda =0$ (cell C). $1-\eta \lambda =1-0=1\Rightarrow w^{\text{new}}=1\times 10=10$ Yeh step kyun? Koi regularizer nahi toh model untouched rehta hai — yahi woh baseline hai jiske baare mein parent ne bataya tha ki overfit karta hai.

Step 2 — $\lambda =2$. $1-(0.1)(2)=0.8\Rightarrow w^{\text{new}}=8$ Yeh step kyun? Healthy shrink: har step mein weight ka $80\%$ bachta hai, toh kaafi steps mein woh smoothly $0$ ki taraf decay karta hai.

Step 3 — $\lambda =20$ (cell D, danger zone). $1-(0.1)(20)=1-2=-1\Rightarrow w^{\text{new}}=-1\times 10=-10$ Yeh step kyun? Shrink factor negative ho gaya. Weight shrink nahi hua — usne sign flip kiya aur size $10$ pe raha! Aur bade $\lambda$ ke saath factor $<-1$ ho jaata hai aur weight explode karta hai. Yahi "$\lambda \to \infty$" ka practical matlab hai: update unstable ho jaata hai jab tak $\eta \lambda <1$ na ho.

Verify: Stability rule hai $|1-\eta \lambda |<1$, yaani $0<\eta \lambda <2$. Check: $\lambda =0\Rightarrow 0$ (edge, koi motion nahi), $\lambda =2\Rightarrow 0.2$ ✓ andar, $\lambda =20\Rightarrow 2.0$ bilkul unstable edge par, jisse humne woh sign flip dekha. Parent ka claim "$\lambda \to \infty$ weights shrink hokar zero ho jaate hain" ideal fixed point hai — lekin sirf tab reachable hai jab $\eta$ itna chhota ho ki $\eta \lambda <2$ rahe. ✓

---

Example 3 — L1 soft-thresholding, small vs large, dono signs (cells E, F)

Forecast: L1 ek fixed amount $\eta \lambda =0.05$ zero ki taraf subtract karta hai. Guess karo kaun se weights bachenge: $0.05$ se chhota weight bach nahi sakta; kaafi bada ek sirf $0.05$ shave off karayega.

Parent ki soft-threshold formula: $w^{\text{new}}=\mathrm{sign}(w)\max (0,|w|-\eta \lambda )$

Step 1 — $w_1=0.05$ (cell E, bilkul threshold par). $|w_1|-\eta \lambda =0.05-0.05=0,\max (0,0)=0\Rightarrow w_1^{\text{new}}=0$ Yeh step kyun? Uski magnitude step ke barabar hai, toh woh bilkul zero par land karta hai aur select out ho jaata hai. Yahi L1 ka automatic feature selection hai action mein.

Step 2 — $w_2=-0.03$ (cell E, negative, below threshold). $|w_2|-\eta \lambda =0.03-0.05=-0.02,\max (0,-0.02)=0\Rightarrow w_2^{\text{new}}=0$ Yeh step kyun? $\max (0,\cdot )$ guard ke bina weight flip ho jaata $-\mathrm{sign}\times$positive — ek bogus overshoot. $\max$ use bilkul zero par clamp karta hai. $\mathrm{sign}$ ne negative input ko sahi handle kiya.

Step 3 — $w_3=1.2$ (cell F, large, survives). $|w_3|-\eta \lambda =1.2-0.05=1.15,\mathrm{sign}(1.2)\times 1.15=+1.15\Rightarrow w_3^{\text{new}}=1.15$ Yeh step kyun? Ek bada informative weight sirf fixed $0.05$ kho deta hai — woh bachta hai aur apna kaam karta rehta hai. L2 se contrast karo, jo proportionally $5\%$ ($=0.06$) le leta.

Verify: Teen mein se do weights bilkul zero ho gaye (sparsity!), bade ne exactly $\eta \lambda =0.05$ shrink kiya. Sanity: absolute weights ka sum $0.05+0.03+1.2=1.28$ se $0+0+1.15=1.15$ gira, drop $0.13$ ka, aur $0.13\le 3\times 0.05=0.15$ (total available step se zyada drop nahi ho sakta). ✓

---

Example 4 — degenerate case $w=0$ exactly (cell G)

Forecast: Parent ne note kiya tha ki $\frac{\partial |w|}{\partial w}$ $w=0$ par undefined hai (diamond ka wahan sharp corner hai). Toh hum kya plug in karein? Guess: penalty "resist" karni chahiye data gradient ko, koi definite direction push karne ki jagah.

Step 1 — zero par subgradient yaad karo. $w=0$ par $|w|$ ka slope ek akela number nahi hai; woh interval $[-1,+1]$ mein koi bhi value ho sakti hai. Yeh set subgradient kehlati hai. Yeh step kyun? Kyunki corner ki kaafi tangent lines hain, ek nahi — hume puri interval se reason karna hoga, ek point se nahi.

Step 2 — check karo ki data gradient penalty se escape kar sakta hai ya nahi. Weight zero chhod sakta hai sirf tab jab data gradient maximum penalty pull $\lambda =0.5$ se zyada strong ho. Yahan $\left|\frac{\partial {\mathcal{L}}_{\text{data}}}{\partial w}\right|=0.3<\lambda =0.5$. Yeh step kyun? Interval $[-\lambda ,\lambda ]=[-0.5,0.5]$ ke andar penalty data gradient ko exactly cancel kar sakti hai subgradient $g^{\ast }=-0.3/0.5\cdot$… choose karke — aur seedha bolein, kyunki $0.3<0.5$, koi subgradient value total gradient ko zero kar deta hai.

Step 3 — conclude karo ki weight same rehta hai. $w^{\text{new}}=0$ Yeh step kyun? L1 se pehle se zeroed weight zeroed hi rehta hai jab tak uska data signal $\lambda$ se zyada na ho. Yahi exactly reason hai ki L1 stable sparsity deta hai — weak signals par features wapas on nahi flickerti.

Verify: Threshold check: activate hone ke liye $|0.3|>0.5$ chahiye; false, toh weight $0$ rehta hai. Agar data gradient instead $0.8>0.5$ hota, toh ek step deta $w^{\text{new}}=-\eta (0.8-\lambda \cdot \mathrm{sign})=-0.1(0.8-0.5)=-0.03$, yaani weight finally zero chhod deta. ✓

---

Example 5 — geometry: kyun L1 diamond zeros banata hai (cell H)

Forecast: Ellipse ko baahri taraf grow karte socho jab tak woh pehli baar budget region ko touch na kare. Woh circle vs diamond ko kahan touch karta hai? Guess: diamond ko pointy corner par touch milta hai.

Step 1 — L2 (circle) side padhho, left panel. Budget $w_1^2+w_2^2\le C$ round region hai. Expanding ellipse (data loss) ise ek smooth arc par touch karti hai, aisa point jahan dono $w_1$ aur $w_2$ nonzero hain. Yeh step kyun? Smooth boundary ki koi preferred direction nahi hoti, toh tangency wahan land hoti hai jahan ellipse lean karti hai — generically off-axis. Result: sab weights chhote, koi exactly zero nahi.

Step 2 — L1 (diamond) side padhho, right panel. Budget $|w_1|+|w_2|\le C$ diamond hai. Uske corners axes par hain (jahan ek coordinate zero hai). Ellipse, expand hote hue, bahut often pehle ek corner se bump karti hai. Yeh step kyun? Corners ellipse ki taraf stick out karte hain aur ellipse orientations ki puri range cover karte hain, toh corner par tangency likely hai. Us corner par ek coordinate exactly zero hai → sparsity.

Step 3 — update rule se connect karo. Yeh geometry Example 3 ki soft-threshold ki same kahani hai: constant push $\eta \lambda$ hi hai jo ek coordinate ko poori tarah axis par drive karta hai.

Verify: Diamond $|w_1|+|w_2|=C$ ka corner $w_2$-axis par $(0,C)$ hai — ek coordinate exactly $0$. Circle $w_1^2+w_2^2=C^2$ ke liye sirf axis points isolated hain; generic tangent line off-axis touch karta hai. ✓ (algebraic check VERIFY mein)

---

Example 6 — dropout expectation, dono scaling conventions (cell I)

Forecast: "Keep probability" $1-p=0.75$ hai. Guess karo expected surviving signal $0.75\times 6=4.5$ hoga classic dropout ke liye, aur inverted dropout engineer kiya gaya hai ki poora $6$ wapas de.

Step 1 — classic dropout, training expectation. $\mathbb{E}[\tilde{h}]=(1-p)h=0.75\times 6.0=4.5$ Yeh step kyun? Average par $75\%$ forward passes neuron ko rakhte hain; mean signal $4.5$ tak scale down ho jaata hai.

Step 2 — classic dropout, test time correction. Test par hum neuron ko hamesha rakhte hain, toh training ke expected $4.5$ se match karne ke liye $(1-p)$ se scale karo: $h^{\text{test}}=(1-p)h=0.75\times 6.0=4.5$ Yeh step kyun? Test ko wahi average magnitude dekhni chahiye jo next layer ko train kiya gaya tha, warna har activation $1/0.75$ zyada bada hoga.

Step 3 — inverted dropout, training scaling. Jab rakha jaaye, $(1-p)$ se divide karo: $\tilde{h}=\frac{h}{1-p}=\frac{6.0}{0.75}=8.0\text{(jab kept ho), }0\text{ (jab dropped ho)}$ $\mathbb{E}[\tilde{h}]=(1-p)\cdot \frac{h}{1-p}+p\cdot 0=h=6.0$ Yeh step kyun? Training ke dauran pre-inflate karke, expectation pehle se poora $h$ hai, toh test time mein koi scaling nahi chahiye — yahi reason hai ki modern frameworks ise use karte hain.

Verify: Classic: train mean $4.5$, test output $4.5$ — matched ✓. Inverted: train mean $=6.0=h$, test output $=6.0$ — matched ✓. Dono conventions identical test behaviour dete hain ($4.5$ effective vs $6.0$ full ek re-parameterization hai; downstream weights ke saath ratio consistent hai). ✓

---

Example 7 — real-world: sahi tool choose karo (cell J)

Forecast: Sparsity + interpretability ek tool chikhti hai; deep net mein overfitting doosra. Guess: sensor-selection linear layer par L1, CNN ke andar dropout.

Step 1 — sensor selection → L1. Kyun? Ex 3 se, L1 weak weights ko exactly zero karta hai, toh 45 useless sensors ka coefficient $0$ ho jaata hai aur model se drop ho jaate hain. Yeh engineers ka "kaun se 5 sensors" wala interpretable jawab deta hai. Feature Engineering se tied.

Step 2 — survivors par smooth stable coefficients → optionally L2. Kyun? Bachne wale 5 sensors mein hum nahi chahte ki koi single coefficient blow up kare (Ex 1 ka proportional decay), toh light L2 unhe well-conditioned rakhti hai.

Step 3 — deep CNN overfitting → dropout. Kyun? Ex 6 se, dropout co-adaptation tod deta hai toh CNN training-flight noise memorize nahi kar sakta. Yeh kaafi sub-networks ko average karne jaisa behave karta hai — Ensemble Methods se related.

Step 4 — $\lambda$ / $p$ honestly choose karo. Kyun? Koi bhi strength a priori nahi pata; inhe Cross-Validation se sweep karo aur train aur test loss ke beech gap dekho (Overfitting Detection).

Verify: Counts par sanity: L1 moderate $\lambda$ ke saath jo 50 mein se 5 features rakhe matlab $\ge 45$ zeros — parent ke Example 2 vector ke 45 zeros ke saath consistent. Dropout $p=0.5$ FC layers par parent ke fault-detection CNN se match karta hai. ✓

---

Example 8 — exam twist: elastic net trap (cell K)

Forecast: Ek saath do forces — proportional L2 shrink aur constant L1 push. Guess: pehle L2 factor apply karo, phir result par L1 soft-threshold.

Combined update (data gradient $0$): $w^{\text{new}}=\mathrm{sign}(w)\max (0,|(1-\eta {\lambda }_2)w|-\eta {\lambda }_1)$

Step 1 — L2 shrink factor apply karo. $1-\eta {\lambda }_2=1-(0.1)(2)=0.8,0.8\times 0.5=0.4$ Yeh step kyun? L2 proportional hai, toh woh pehle multiplicatively act karta hai, deta hai $0.4$.

Step 2 — us result par L1 soft-threshold apply karo. $\eta {\lambda }_1=(0.1)(1)=0.1,\max (0,0.4-0.1)=0.3,\mathrm{sign}(0.5)\times 0.3=0.3$ Yeh step kyun? Ab constant L1 step subtract karo. Dono pulls zero ki taraf, lekin akele kisi ne bhi zero reach nahi kiya, toh weight $0.3$ par bachta hai.

Step 3 — trap spot karo. Agar tum (galat se) L1 pehle apply karo: $\max (0,0.5-0.1)=0.4$, phir $\times 0.8=0.32$. Alag jawab! Convention hai L2 shrink phir L1 threshold, deta hai $0.3$. Yeh step kyun? Kyunki L1 $\max (0,\cdot )$ nonlinear hai, order matter karta hai — classic exam gotcha.

Verify: Correct order deta hai $0.30$; galat order deta hai $0.32$; woh differ karte hain, order-sensitivity confirm karte hue. Dono weight ko $0.5$ se neeche pull karte hain (elastic net = "L1 sparsity flavor + L2 stability"). ✓

---

Recall Self-test (jawab dene ke baad reveal karo)

$w=4$ par L2 step, data grad $1$, $\eta =0.1$, $\lambda =0.5$ kya deta hai? ::: $3.7$ L1 step $w=0.05$ ko zero karta hai jab $\eta \lambda$ kis value ke barabar ho? ::: $0.05$ L2 stability ke liye, $\eta \lambda$ kya se neeche rehna chahiye? ::: $2$ Inverted dropout with $p=0.25$: value $6$ ka ek kept neuron kya ban jaata hai? ::: $8.0$ Elastic-net order (kaun si penalty pehle)? ::: L2 shrink, phir L1 threshold

Related deep-dive links: Gradient Descent Variants (kaise $\eta$ in updates ke saath interact karta hai), Neural Network Architectures (kahan dropout layers baithti hain), Bayesian Inference (L2 = Gaussian prior, L1 = Laplace prior — probabilistic story).