Regularization — L1 (lasso), L2 (ridge), dropout

5.6.3 · Coding › Machine Learning (Aerospace Applications)

YE KAAM KYUN KARTA HAI: Overfitting tab hoti hai jab model ki capacity data ke liye bahut zyada hoti hai. Regularization artificially us capacity ko constrain karta hai bade weights ko penalize karke, effectively model ki noise fit karne ki ability reduce karta hai.

Teen core techniques:

• L2 (Ridge): Weights ki squared magnitude ko penalize karo → smooth, small weights
• L1 (Lasso): Absolute magnitude ko penalize karo → sparse weights (bahut saare exactly zero)
• Dropout: Training ke dauran randomly neurons ko zero kar do → robust features jo kisi ek single pathway par rely nahi kar sakti

---

L2 Regularization (Ridge)

FIRST PRINCIPLES SE DERIVATION:

Standard empirical risk minimization se shuru karo: ${\mathcal{L}}_{\text{data}}=\frac{1}{N}{\sum }_{n=1}^N\ell (y_n,{\hat{y}}_n)$

Overfitting rokne ke liye, hum weights ko small rakhna chahte hain. Squared kyun? Do reasons:

1. Har jagah differentiable (absolute value ke unlike, jiska zero par ek kink hota hai)
2. Quadratic penalty bade weights ke liye quickly badhta hai lekin chhote weights ke liye gentle hai

Penalty add karo: ${\mathcal{L}}_{\text{total}}=\frac{1}{N}{\sum }_{n=1}^N\ell (y_n,{\hat{y}}_n)+\frac{\lambda }{2}\Vert w{\Vert }_2^2$

$\frac{1}{2}$ conventional hai (derivative ke saath cancel ho jaata hai). $\lambda$ trade-off control karta hai:

• $\lambda =0$: koi regularization nahi, overfitting ka risk
• $\lambda \to \infty$: weights zero ho jaate hain, model useless ho jaata hai

Gradient update derivation: $\frac{\partial {\mathcal{L}}_{\text{total}}}{\partial w_i}=\frac{\partial {\mathcal{L}}_{\text{data}}}{\partial w_i}+\lambda w_i$

Gradient descent ke dauran: $w_i\leftarrow w_i-\eta \left(\frac{\partial {\mathcal{L}}_{\text{data}}}{\partial w_i}+\lambda w_i\right)$

Rearrange karo: $w_i\leftarrow (1-\eta \lambda )w_i-\eta \frac{\partial {\mathcal{L}}_{\text{data}}}{\partial w_i}$

YE STEP KYUN? $(1-\eta \lambda )$ term weight decay hai—har update weight ko data gradient apply karne se pehle thoda shrink karta hai. Isliye L2 regularization ko "weight decay" bhi kehte hain.

Model: Linear regression $C_D=w_1\alpha +w_{2}M+w_{3}Re+w_4{M}^2+w_5{\alpha }^2+b$

Regularization ke bina: 100 wind tunnel runs par training karne se milta hai: $w=[0.05,2.1,-0.003,8.4,0.12]$ Bada $w_4=8.4$ predictions ko Mach number ke liye extremely sensitive bana deta hai—chhoti si measurement errors bahut bade swings cause karti hain.

L2 regularization ke saath ($\lambda =0.1$): $w=[0.048,1.2,-0.00002,1.8,0.09]$

YE STEP KYUN? Loss mein penalty $\frac{0.1}{2}(0.048^2+1.2^2+\dots )$ saare weights ko smaller push karta hai. Model thodi si training accuracy sacrifice karta hai (shayad RMSE 0.02 se 0.025 ho jaaye) lekin nayi flight conditions par bahut better generalize karta hai.

Result: Test error 0.08 se 0.03 ho jaata hai. Model seekhta hai ki $M^2$ matter karta hai, lekin absurdly bade coefficient ke saath nahi.

---

L1 Regularization (Lasso)

FIRST PRINCIPLES SE DERIVATION:

Squared ki jagah absolute value kyun? Sparsity.

Constraint view consider karo: ${\mathcal{L}}_{\text{data}}$ minimize karo subject to $\Vert w{\Vert }_1\le C$. 2D mein:

• L2 constraint $w_1^2+w_2^2\le C$ ek circle hai
• L1 constraint $|w_1|+|w_2|\le C$ ek sharp corners wala diamond hai

Jab loss function ke contours (ellipses) constraint region ko touch karte hain, toh wo corner par touch karne ki likely hoti hai jahan ek coordinate exactly zero hoti hai. Isliye L1 sparse solutions produce karta hai.

L1 ka Gradient (subgradient):

+1 & w_i > 0 \\ -1 & w_i < 0\\ \text{undefined} & w_i = 0 \end{cases} = \text{sign}(w_i)$$ Update: $$w_i \leftarrow w_i - \eta \left( \frac{\partial \mathcal{L}_{\text{data}}}{\partial w_i} + \lambda \cdot \text{sign}(w_i) \right)$$ **YE STEP KYUN?** L2 ke $(1 - \eta\lambda)$ proportional shrinkage ke unlike, L1 ek **constant** $\eta \lambda$ subtract karta hai. Chhote weights zero cross karke poori tarah zero ho jaate hain. Bade weights fixed amount se shrink hote hain lekin rarely exactly zero tak pahunchte hain. > [!formula] Weights par L1 ka Effect > Chhote $|w_i| < \frac{\eta\lambda}{1}$ ke liye, weight exactly zero set ho jaata hai (soft thresholding). > $$w_i^{\text{new}} = \text{sign}(w_i) \max(0, |w_i| - \eta\lambda)$$ > [!example] Example 2: Sensor Fusion ke liye Lasso > **Task:** 50 engine sensors se thrust predict karna. Sirf ~5 sensors real signal carry karte hain; baaki redundant ya noisy hain. **Regularization ke bina:** Model saare 50 features tiny weights ke saath use karta hai. Slow chalta hai, overfit karta hai. **L1 ke saath ($\lambda = 0.01$):** Training ke baad, weights aisa dikhte hain: $$w = [0, 0, 1.2, 0, 0, -0.8, 0, \ldots, 0, 0.5, 0]$$ **YE STEP KYUN?** L1 penalty ne 45 weights ko exactly zero kar diya. Model ==automatic feature selection== perform karta hai, sirf EGT (exhaust gas temp), N2 speed, aur fuel flow rate ko rakhta hai. **Result:** Model simpler, faster, aur better generalizing hai. Interpretability improve hoti hai—engineers dekh sakte hain ki kaun se sensors matter karte hain. --- ## Dropout > [!definition] Dropout > Training ke dauran, har neuron ka output probability $p$ se randomly zero karo (typically $p=0.5$ hidden layers ke liye). Test time par, saare neurons use karo lekin outputs ko $(1-p)$ se scale karo. **FIRST PRINCIPLES SE DERIVATION:** **YE KAAM KYUN KARTA HAI?** Traditional neural networks ==co-adaptations== develop kar sakte hain: neuron A sirf tab fire karta hai jab neuron B fire kare, brittle, interdependent features create karte hain. Agar B ka input thoda change ho, toh poora pathway break ho jaata hai. Dropout har neuron ko **independently** kaam karne par majboor karta hai. Kyunki koi bhi neuron drop ho sakta hai, features ko apne aap useful hona padta hai. Ye ek ensemble effect create karta hai: har training iteration ek alag sub-network use karta hai, aur final model ~$2^n$ implicit sub-networks ka average karta hai (jahan $n$ neurons ki sankhya hai). **Mathematical view:** Maano $h_i$ ek hidden neuron ka activation hai. Dropout ke saath: $$\tilde{h}_i = \begin{cases} 0 & \text{with probability } p \\ h_i & \text{with probability } 1-p \end{cases}$$ Expectation mein: $\mathbb{E}[\tilde{h}_i] = (1-p) h_i$ **YE STEP KYUN?** Test time par, hum dropout nahi karte (hum full network ki power chahte hain), lekin hume expected activation match karni padti hai. Hum $(1-p)$ se scale karte hain: $$h_i^{\text{test}} = (1-p) \cdot h_i$$ Isse modern frameworks mein **inverted dropout** kehte hain—wo training ke dauran $\frac{1}{1-p}$ se scale karte hain **instead**: $$\tilde{h}_i = \begin{cases} 0 & \text{with probability } p \\ \frac{h_i}{1-p} & \text{with probability } 1-p \end{cases}$$ Ab $\mathbb{E}[\tilde{h}_i] = h_i$, aur test time par koi scaling nahi chahiye. > [!formula] Dropout Training vs Test > **Training:** > $$\tilde{h}_i = m_i \cdot h_i / (1-p), \quad m_i \sim \text{Bernoulli}(1-p)$$ > **Test:** > $$h_i^{\text{test}} = h_i \quad \text{(no mask, no scaling)}$$ > [!example] Example 3: Aircraft Fault Detection CNN mein Dropout > **Task:** Sensor time-series ko "normal" vs 8 fault modes mein classify karna. Network: 3 conv layers + 2 fully-connected layers. **Dropout ke bina:** 50 epochs ke baad, training accuracy = 99%, test accuracy = 78%. Network ne training flights mein noise patterns memorize kar liye. **Dropout ke saath ($p=0.5$ FC layers par):** 50 epochs ke baad: training accuracy = 92%, test accuracy = 89%. **YE STEP KYUN?** Dropout ne FC layers ko specific training examples memorize karne se roka. Har forward pass ek alag sub-network use karta tha, isliye model ko aise robust features seekhne padte the jo kai configurations mein kaam karein. **Implementation detail:** Dropout training ke dauran **per-batch** apply hota hai. Har mini-batch ek alag random mask dekhta hai. **Result:** Model unseen aircraft aur flight regimes par generalize karta hai. Interpretability: ek time mein ek layer se dropout hataane se pata chala ki second FC layer overfitting ke liye sabse zyada prone thi. --- ## L1, L2, aur Dropout ki Comparison | **Technique** | **Weights par Effect** | **Best Use Case** | **Computational Cost** | |------------|----------------------|-------------------|----------------------| | **L2 (Ridge)** | Saare weights smoothly shrink hote hain | General-purpose, hamesha pehle try karo | Negligible (gradient mein ek extra term) | | **L1 (Lasso)** | Bahut saare weights → exactly 0 | High-dimensional jahan irrelevant features hain | Thoda zyada (subgradient) | | **Dropout** | Direct weight penalty nahi, lekin co-adaptation reduce karta hai | Deep networks jo overfitting prone hain | ~2× training time (zyada epochs chahiye) | **YE DIFFERENCES KYUN?** - **L2** default hai kyunki ye smooth hai, tune karna easy hai, aur almost har jagah kaam karta hai. - **L1** tab hai jab tumhare paas sparsity prior ho—tum mante ho ki most features useless hain. - **Dropout** deep networks ke liye hai jahan co-adaptation problem hai, sirf bade weights nahi. **Inhe combine karna:** L2 + dropout saath use karna common hai. L2 weight magnitudes handle karta hai, dropout co-adaptation handle karta hai. L1 + L2 saath ko ==Elastic Net== kehte hain. > [!mistake] Common Mistake: Test Time par Dropout Apply Karna > **Galat idea:** "Dropout model ko better banata hai, toh main ise test time par bhi use karunga." **Ye sahi kyun lagta hai:** Tum dekhte ho ki dropout on ke saath higher training accuracy hai, toh ye ek acchi cheez lagti hai. **Fix:** Dropout ek **training-only** technique hai. Test time par: - Tum saare neurons ki full power chahte ho (tum ab overfitting rokne ki koshish nahi kar rahe) - Random zeroing predictions ko non-deterministic banaa dega—same input do baar run karo, alag outputs milenge - $(1-p)$ scaling ensure karta hai ki expected output match kare **Steel-man:** Confusion isliye hoti hai kyunki dropout *model ko improve* karta hai—lekin improvement noise ke saath training se aati hai, inference par noise add karne se nahi. Socho aise jaise weighted vest ke saath weight lifting: tum training ke dauran vest pehnte ho strong hone ke liye, lekin actual competition ke liye utaar dete ho. > [!mistake] Common Mistake: $\lambda$ Bahut High Set Karna > **Galat idea:** "Zyada regularization = better generalization, toh $\lambda$ badha do." **Ye sahi kyun lagta hai:** Test error $\lambda$ ko 0 se badhane par decrease hoti hai... toh continue karo! **Fix:** Ek U-shaped curve hai: - $\lambda$ bahut chhota: overfitting (high test error) - $\lambda$ bahut bada: underfitting (model data patterns nahi seekh sakta, high test error) **YE STEP KYUN?** Optimal $\lambda$ dhundhne ke liye cross-validation use karo. Typical search: $\lambda \in [10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}, 1]$ log scale par. **Example:** Drag coefficient model ke liye, $\lambda = 0.1$ optimal tha. $\lambda = 1.0$ ne saare weights ko near zero force kar diya, aur model ne har cheez ke liye sirf mean drag coefficient predict kiya (useless). --- > [!recall]- Ek 12-Saal Ke Bacche Ko Samjhao > Socho tum ek test ke liye padh rahe ho, lekin tumhare paas 1000 notes ki ek cheat sheet hai. Tum *could* har single note memorize karo, including typos aur random doodles. Lekin jab real test aata hai (thode alag questions ke saath), tum fail ho jaoge kyunki tumne nonsense yaad kiya understanding ki jagah. **Regularization ek rule ki tarah hai:** "Tum apni cheat sheet par sirf 10 cheezein likh sakte ho, aur wo short honi chahiye." Ab tum sirf sabse important concepts likhne par majboor ho—wo cheezein jo real test par actually help karein. - **L2** kehta hai: "Apne notes short rakho." (Saare notes chhote ho jaate hain, koi disappear nahi hota.) - **L1** kehta hai: "Tumhare paas total sirf 10 notes ho sakte hain, baaki blank hone chahiye." (Zyaadatar notes gayab ho jaate hain, kuch survive karte hain.) - **Dropout** kehta hai: "Har baar jab tum padho, randomly apni cheat sheet ka aadha hissa cover kar lo." (Tum har concept ko independently samajhna seekhte ho, ye rely nahi karte ki note #5 hamesha note #6 ke paas ho.) Jab real test aata hai, tum ready ho kyunki tumne *ideas* seekhe, memorization tricks nahi. --- > [!mnemonic] LASSO Losers Ko Lasso Karta Hai > **L**1 → **LASSO** → Lassoes (rope se kheench ke zero par laata hai) un **loser** features ko jo contribute nahi karte. **L**2 → **L**ess extreme → **S**moothly **S**hrinkata hai (dono S se shuru hote hain), saare weights ko gently shrink karta hai. **Dropout** → **Drop**out of school → tum kabhi kabhi **bahar** hote ho, toh tum **akele** survive karna seekhte ho (neurons independently kaam karte hain). --- ## Connections - [[Bias-Variance Tradeoff]: Regularization variance reduce karta hai thodi si bias ki cost par - [[Cross-Validation]]: $\lambda$ hyperparameter tune karne ke liye use hota hai - [[Gradient Descent Variants]]: L2 update rule mein ek decay term add karta hai - [[Feature Engineering]]: L1 automatic feature selection perform karta hai - [[Ensemble Methods]]: Dropout sub-networks ka ek implicit ensemble create karta hai - [[Overfitting Detection]]: Learning curves dikhate hain jab regularization ki zaroorat hai - [[Neural Network Architectures]]: Dropout typically fully-connected layers par apply hota hai, convolutional layers par nahi - [[Bayesian Inference]]: Regularization weights par ek prior distribution correspond karta hai (L2 = Gaussian prior, L1 = Laplace prior) --- #flashcards/coding Regularization ke peeche core idea kya hai? :: Loss function mein ek penalty term add karo jo model complexity ko discourage kare, model ko noise memorize karne ki jagah sirf robust, generalizable patterns seekhne par majboor kare. L2 regularization penalty term kya hai? ::: $\frac{\lambda}{2} \sum_i w_i^2$, squared weights ka sum jo regularization strength $\lambda$ se scale hota hai. L2 regularization ko "weight decay" kyun kehte hain? ::: Kyunki gradient update ban jaata hai $w_i \leftarrow (1 - \eta\lambda) w_i - \eta \nabla \mathcal{L}_{\text{data}}$, jahan $(1-\eta\lambda)$ term weights ko har step par ek small factor se decay karta hai. L1 aur L2 regularization mein key difference kya hai? ::: L1 (Lasso) $\sum_i |w_i|$ penalize karta hai aur sparse solutions produce karta hai (bahut saare weights exactly zero). L2 (Ridge) $\sum_i w_i^2$ penalize karta hai aur saare weights ko smoothly shrink karta hai lekin rarely exactly zero tak. L1 regularization sparse weights kyun produce karta hai? ::: L1 constraint $|w_1| + |w_2| \leq C$ ek diamond shape banata hai sharp corners ke saath. Jab loss contours constraint ko touch karte hain, wo corners par hit karte hain jahan ek coordinate exactly zero hoti hai. Dropout kya hai aur kab apply hota hai? ::: Training ke dauran sirf neuron outputs ko probability $p$ se randomly zero karo. Test time par, saare neurons use karo (no dropout). Ye co-adaptation rokta hai aur ensemble effect create karta hai. Training ke dauran inverted dropout formula kya hai? ::: $\tilde{h}_i = m_i \cdot h_i / (1-p)$ jahan $m_i \sim \text{Bernoulli}(1-p)$. Ye active neurons ko scale up karta hai taaki expected activation test time se match kare. Test time par dropout kyun apply NAHI karna chahiye? ::: Dropout training-only hai. Test time par tum chahte ho (1) full network power, (2) deterministic predictions, aur (3) expected activation values jo training expectations se match karein. Agar $\lambda$ bahut high set ho toh kya hoga? ::: Model underfit karta hai—weights itne chhote ho jaate hain ki data patterns capture nahi kar sakte, aur model input ki parwaah kiye bina sirf mean output predict karta hai. Optimal $\lambda$ kaise dhundhte hain? ::: Cross-validation use karo log-scale grid search ke saath, e.g., $\lambda \in [10^{-5}, 10^{-4}, \ldots, 1]$, aur wo value choose karo jisme lowest validation error ho. L1 regularization ka main use case kya hai? ::: High-dimensional problems jahan tum mante ho ki most features irrelevant hain—L1 automatic feature selection perform karta hai irrelevant weights ko exactly zero drive karke. Elastic net regularization kya hai? ::: L1 aur L2 ka combination: $\lambda_1 \sum_i |w_i| + \lambda_2 \sum_i w_i^2$. Dono sparsity (L1) aur smooth shrinkage (L2) milte hain. Dropout neurons ko independently kaam karne par kyun majboor karta hai? ::: Kyunki koi bhi neuron randomly drop ho sakta hai, features ko apne aap useful hona padta hai—wo specific doosre neurons ke saath co-adapt karne par rely nahi kar sakte jo absent ho sakte hain. L2 regularization ki Bayesian interpretation kya hai? ::: Regularization weights par ek Gaussian prior lagane ke corresponding hai: $p(w) \propto \exp(-\frac{\lambda}{2} w^2)$. Regularized likelihood maximize karna = MAP estimation. Aerospace ML mein regularization critical kyun hai? ::: Flight data expensive aur limited hota hai. Regularization ke bina, models training flights memorize kar lete hain (sensor noise aur anomalies including) aur nayi flight conditions par catastrophically fail ho jaate hain. ## 🖼️ Concept Map ```mermaid flowchart TD OF[Overfitting: model fits noise] REG[Regularization: penalty on complexity] L2[L2 Ridge: sum squared weights] L1[L1 Lasso: sum absolute weights] DROP[Dropout: randomly zero neurons] LAM[Lambda: reg strength] SMOOTH[Small smooth weights] SPARSE[Sparse weights, many zero] ROBUST[Robust features] DECAY[Weight decay: 1 minus eta lambda] OF -->|solved by| REG REG -->|technique| L2 REG -->|technique| L1 REG -->|technique| DROP LAM -->|controls tradeoff| L2 LAM -->|controls tradeoff| L1 L2 -->|differentiable, gentle| SMOOTH L1 -->|kink at zero| SPARSE DROP -->|no single pathway| ROBUST L2 -->|gradient yields| DECAY DECAY -->|shrinks weights each step| SMOOTH ```