Regularization — L1 (lasso), L2 (ridge), dropout

WHY it works: Overfitting happens when the model has too much capacity for the data. Regularization artificially constrains that capacity by penalizing large weights, effectively reducing the model's ability to fit noise.

Three core techniques:

• L2 (Ridge): Penalize the squared magnitude of weights → smooth, small weights
• L1 (Lasso): Penalize the absolute magnitude → sparse weights (many exactly zero)
• Dropout: Randomly zero out neurons during training → robust features that can't rely on any single pathway

---

L2 Regularization (Ridge)

DERIVATION FROM FIRST PRINCIPLES:

Start with standard empirical risk minimization: $\mathcal{L}_{\text{data}} = \frac{1}{N} \sum_{n=1}^{N} \ell(y_n, \hat{y}_n)$

To prevent overfitting, we want to keep weights small. Why squared? Two reasons:

1. Differentiable everywhere (unlike absolute value, which has a kink at zero)
2. Quadratic penalty grows quickly for large weights but is gentle for small weights

Add the penalty: $\mathcal{L}_{\text{total}} = \frac{1}{N} \sum_{n=1}^{N} \ell(y_n, \hat{y}_n) + \frac{\lambda}{2} \|w\|_2^2$

The $\frac{1}{2}$ is conventional (cancels with the derivative). The $\lambda$ controls the trade-off:

• $\lambda = 0$: no regularization, risk overfitting
• $\lambda \to \infty$: weights shrink to zero, model becomes useless

Gradient update derivation: $\frac{\partial \mathcal{L}_{\text{total}}}{\partial w_i} = \frac{\partial \mathcal{L}_{\text{data}}}{\partial w_i} + \lambda w_i$

During gradient descent: $w_i \leftarrow w_i - \eta \left( \frac{\partial \mathcal{L}_{\text{data}}}{\partial w_i} + \lambda w_i \right)$

Rearranging: $w_i \leftarrow (1 - \eta \lambda) w_i - \eta \frac{\partial \mathcal{L}_{\text{data}}}{\partial w_i}$

WHY THIS STEP? The $(1 - \eta \lambda)$ term is weight decay—each update shrinks the weight slightly before applying the data gradient. This is why L2 regularization is also called "weight decay."

Model: Linear regression $C_D = w_1 \alpha + w_2 M + w_3 Re + w_4 M^2 + w_5 \alpha^2 + b$

Without regularization: Training on 100 wind tunnel runs gives: $w = [0.05, 2.1, -0.003, 8.4, 0.12]$ The large $w_4 = 8.4$ makes predictions extremely sensitive to Mach number—tiny measurement errors cause huge swings.

With L2 regularization ($\lambda = 0.1$): $w = [0.048, 1.2, -0.00002, 1.8, 0.09]$

WHY THIS STEP? The penalty $\frac{0.1}{2}(0.048^2 + 1.2^2 + \ldots)$ in the loss pushes all weights smaller. The model sacrifices a bit of training accuracy (maybe RMSE goes from 0.02 to 0.025) but generalizes far better to new flight conditions.

Result: Test error drops from 0.08 to 0.03. The model learns that $M^2$ matters, but not with an absurdly large coefficient.

---

L1 Regularization (Lasso)

DERIVATION FROM FIRST PRINCIPLES:

Why absolute value instead of squared? Sparsity.

Consider the constraint view: minimize $\mathcal{L}_{\text{data}}$ subject to $\|w\|_1 \leq C$. In 2D:

• L2 constraint $w_1^2 + w_2^2 \leq C$ is a circle
• L1 constraint $|w_1| + |w_2| \leq C$ is a diamond with sharp corners

When the loss function's contours (ellipses) touch the constraint region, they're likely to touch at a corner where one coordinate is exactly zero. This is why L1 produces sparse solutions.

Gradient (subgradient) of L1:

+1 & w_i > 0 \\ -1 & w_i < 0\\ \text{undefined} & w_i = 0 \end{cases} = \text{sign}(w_i)$$ Update: $$w_i \leftarrow w_i - \eta \left( \frac{\partial \mathcal{L}_{\text{data}}}{\partial w_i} + \lambda \cdot \text{sign}(w_i) \right)$$ **WHY THIS STEP?** Unlike L2's $(1 - \eta\lambda)$ proportional shrinkage, L1 subtracts a **constant** $\eta \lambda$. Small weights cross zero and get zeroed out entirely. Large weights shrink by a fixed amount but rarely reach exactly zero. > [!formula] L1 Effect on Weights > For small $|w_i| < \frac{\eta\lambda}{1}$, the weight is set to exactly zero (soft thresholding). > $$w_i^{\text{new}} = \text{sign}(w_i) \max(0, |w_i| - \eta\lambda)$$ > [!example] Example 2: Lasso for Sensor Fusion > **Task:** Predict thrust from 50 engine sensors. Only ~5 sensors carry real signal; the rest are redundant or noisy. **Without regularization:** Model uses all 50 features with tiny weights. Runs slow, overfits. **With L1 ($\lambda = 0.01$):** After training, weights look like: $$w = [0, 0, 1.2, 0, 0, -0.8, 0, \ldots, 0, 0.5, 0]$$ **WHY THIS STEP?** The L1 penalty drove 45 weights to exactly zero. The model performs ==automatic feature selection==, keeping only EGT (exhaust gas temp), N2 speed, and fuel flow rate. **Result:** Model is simpler, faster, and generalizes better. Interpretability improves—engineers see which sensors matter. --- ## Dropout > [!definition] Dropout > During training, randomly set each neuron's output to zero with probability $p$ (typically $p=0.5$ for hidden layers). At test time, use all neurons but scale outputs by $(1-p)$. **DERIVATION FROM FIRST PRINCIPLES:** **WHY does this work?** Traditional neural networks can develop ==co-adaptations==: neuron A only fires when neuron B fires, creating britle, interdependent features. If B's input changes slightly, the whole pathway breaks. Dropout forces each neuron to work **independently**. Since any neuron can be dropped, features must be useful on their own. This creates an ensemble effect: each training iteration uses a different sub-network, and the final model averages over ~$2^n$ implicit sub-networks (where $n$ is the number of neurons). **Mathematical view:** Let $h_i$ be a hidden neuron's activation. With dropout: $$\tilde{h}_i = \begin{cases} 0 & \text{with probability } p \\ h_i & \text{with probability } 1-p \end{cases}$$ In expectation: $\mathbb{E}[\tilde{h}_i] = (1-p) h_i$ **WHY THIS STEP?** At test time, we don't dropout (we want the full network's power), but we must match the expected activation. We scale by $(1-p)$: $$h_i^{\text{test}} = (1-p) \cdot h_i$$ This is called **inverted dropout** in modern frameworks—they scale **during training** by $\frac{1}{1-p}$ instead: $$\tilde{h}_i = \begin{cases} 0 & \text{with probability } p \\ \frac{h_i}{1-p} & \text{with probability } 1-p \end{cases}$$ Now $\mathbb{E}[\tilde{h}_i] = h_i$, and test time needs no scaling. > [!formula] Dropout Training vs Test > **Training:** > $$\tilde{h}_i = m_i \cdot h_i / (1-p), \quad m_i \sim \text{Bernoulli}(1-p)$$ > **Test:** > $$h_i^{\text{test}} = h_i \quad \text{(no mask, no scaling)}$$ > [!example] Example 3: Dropout in Aircraft Fault Detection CNN > **Task:** Classify sensor time-series into "normal" vs 8 fault modes. Network: 3 conv layers + 2 fully-connected layers. **Without dropout:** After50 epochs, training accuracy = 99%, test accuracy = 78%. The network memorized noise patterns in the training flights. **With dropout ($p=0.5$ on FC layers):** After 50 epochs: training accuracy = 92%, test accuracy = 89%. **WHY THIS STEP?** Dropout prevented the FC layers from memorizing specific training examples. Each forward pass used a different sub-network, so the model had to learn robust features that work across many configurations. **Implementation detail:** Dropout is applied **per-batch** during training. Each mini-batch sees a different random mask. **Result:** The model generalizes to unseen aircraft and flight regimes. Interpretability: removing dropout from one layer at a time revealed that the second FC layer was most prone to overfitting. --- ## Comparing L1, L2, and Dropout | **Technique** | **Effect on Weights** | **Best Use Case** | **Computational Cost** | |------------|----------------------|-------------------|----------------------| | **L2 (Ridge)** | All weights shrink smoothly | General-purpose, always try first | Negligible (one extra term in gradient) | | **L1 (Lasso)** | Many weights → exactly0 | High-dimensional with irrelevant features | Slightly higher (subgradient) | | **Dropout** | No direct weight penalty, but reduces co-adaptation | Deep networks prone to overfitting | ~2× training time (need more epochs) | **WHY THESE DIFFERENCES?** - **L2** is the default because it's smooth, easy to tune, and works almost everywhere. - **L1** is for when you have a sparsity prior—you believe most features are useless. - **Dropout** is for deep networks where co-adaptation is the problem, not just large weights. **Combining them:** Common to use L2 + dropout together. L2 handles weight magnitudes, dropout handles co-adaptation. L1 + L2 together is called ==Elastic Net==. > [!mistake] Common Mistake: Applying Dropout at Test Time > **Wrong idea:** "Dropout makes the model better, so I'll use it at test time too." **Why it feels right:** You see higher training accuracy with dropout on, so it seems like a good thing. **The fix:** Dropout is a **training-only** technique. At test time: - You want the full power of all neurons (you're not trying to prevent overfitting anymore) - Random zeroing would make predictions non-deterministic—run the same input twice, get different outputs - The $(1-p)$ scaling ensures the expected output matches **Steel-man:** The confusion arises because dropout *does* improve the model—but the improvement comes from training with noise, not from adding noise at inference. Think of it like lifting weights with a weighted vest: you wear the vest during training to get stronger, but you take it off for the actual competition. > [!mistake] Common Mistake: Setting $\lambda$ Too High > **Wrong idea:** "More regularization = better generalization, so crank $\lambda$ up." **Why it feels right:** Test error decreases as you increase $\lambda$ from 0... so keep going! **The fix:** There's a U-shaped curve: - $\lambda$ too small: overfitting (high test error) - $\lambda$ too large: underfitting (model can't learn the data, high test error) **WHY THIS STEP?** Use cross-validation to find the optimal $\lambda$. Typical search: $\lambda \in [10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}, 1]$ on a log scale. **Example:** For the drag coefficient model, $\lambda = 0.1$ was optimal. $\lambda = 1.0$ forced all weights near zero, and the model just predicted the mean drag coefficient for everything (useless). --- > [!recall]- Explain to a 12-Year-Old > Imagine you're studying for a test, but you have a cheat sheet with 1000 notes. You *could* memorize every single note, including typos and random doodles. But when the real test comes (with slightly different questions), you'll fail because you memorized nonsense instead of understanding. **Regularization is like a rule:** "You can only write10 things on your cheat sheet, and they have to be short." Now you're forced to write only the most important concepts—the stuff that will actually help on the real test. - **L2** says: "Keep your notes short." (All notes get shorter, none disappear.) - **L1** says: "You can only have 10 notes total, the rest must be blank." (Most notes vanish, a few survive.) - **Dropout** says: "Every time you study, randomly cover up half your cheat sheet." (You learn to understand each concept independently, not relying on always having note #5 next to note #6.) When the real test comes, you're ready because you learned the *ideas*, not the memorization tricks. --- > [!mnemonic] LASSO Lassoes Losers > **L**1 → **LASSO** → Lassoes (ropes and pulls to zero) the **loser** features that don't contribute. **L**2 → **L**ess extreme → **S**mooths **S**lowly (both start with S), shrinks all weights gently. **Dropout** → **Drop**out of school → you're **out** sometimes, so you learn to survive **alone** (neurons work independently). --- ## Connections - [[Bias-Variance Tradeoff]: Regularization reduces variance at the cost of slight bias - [[Cross-Validation]]: Used to tune $\lambda$ hyperparameter - [[Gradient Descent Variants]]: L2 adds a decay term to the update rule - [[Feature Engineering]]: L1 performs automatic feature selection - [[Ensemble Methods]]: Dropout creates an implicit ensemble of sub-networks - [[Overfitting Detection]]: Learning curves show when regularization is needed - [[Neural Network Architectures]]: Dropout is typically applied to fully-connected layers, not convolutional layers - [[Bayesian Inference]]: Regularization corresponds to a prior distribution on weights (L2 = Gaussian prior, L1 = Laplace prior) --- #flashcards/coding What is the core idea behind regularization? :: Add a penalty term to the loss function that discourages model complexity, forcing the model to learn only robust, generalizable patterns instead of memorizing noise. What is the L2 regularization penalty term? ::: $\frac{\lambda}{2} \sum_i w_i^2$, the sum of squared weights scaled by regularization strength $\lambda$. Why is L2 regularization also called "weight decay"? ::: Because the gradient update becomes $w_i \leftarrow (1 - \eta\lambda) w_i - \eta \nabla \mathcal{L}_{\text{data}}$, where the $(1-\eta\lambda)$ term decays weights by a small factor each step. What is the key difference between L1 and L2 regularization? ::: L1 (Lasso) penalizes $\sum_i |w_i|$ and produces sparse solutions (many weights exactly zero). L2 (Ridge) penalizes $\sum_i w_i^2$ and shrinks all weights smoothly but rarely to exactly zero. Why does L1 regularization produce sparse weights? ::: The L1 constraint $|w_1| + |w_2| \leq C$ forms a diamond shape with sharp corners. When the loss contours touch the constraint, they hit corners where one coordinate is exactly zero. What is dropout and when is it applied? ::: Randomly set neuron outputs to zero with probability $p$ during training only. At test time, use all neurons (no dropout). This prevents co-adaptation and creates an ensemble effect. What is the inverted dropout formula during training? ::: $\tilde{h}_i = m_i \cdot h_i / (1-p)$ where $m_i \sim \text{Bernoulli}(1-p)$. This scales up active neurons so expected activation matches test time. Why should you NOT apply dropout at test time? ::: Dropout is training-only. At test time you want (1) full network power, (2) deterministic predictions, and (3) the expected activation values that match training expectations. What happens if $\lambda$ is set too high? ::: The model underfits—weights are forced too small to capture the data patterns, and the model just predicts the mean output regardless of input. How do you find the optimal $\lambda$? ::: Use cross-validation with a log-scale grid search, e.g., $\lambda \in [10^{-5}, 10^{-4}, \ldots, 1]$, and pick the value with lowest validation error. What is the main use case for L1 regularization? ::: High-dimensional problems where you believe most features are irrelevant—L1 performs automatic feature selection by driving irrelevant weights to exactly zero. What is elastic net regularization? ::: Combination of L1 and L2: $\lambda_1 \sum_i |w_i| + \lambda_2 \sum_i w_i^2$. Gets both sparsity (L1) and smooth shrinkage (L2). Why does dropout force neurons to work independently? ::: Since any neuron can be randomly dropped, features must be useful on their own—they can't rely on co-adapting with specific other neurons that might be absent. What is the Bayesian interpretation of L2 regularization? ::: Lization corresponds to placing a Gaussian prior on the weights: $p(w) \propto \exp(-\frac{\lambda}{2} w^2)$. Maximizing regularized likelihood = MAP estimation. In aerospace ML, why is regularization critical? ::: Flight data is expensive and limited. Without regularization, models memorize training flights (including sensor noise and anomalies) and fail catastrophically on new flight conditions. ## 🖼️ Concept Map ```mermaid flowchart TD OF[Overfitting: model fits noise] REG[Regularization: penalty on complexity] L2[L2 Ridge: sum squared weights] L1[L1 Lasso: sum absolute weights] DROP[Dropout: randomly zero neurons] LAM[Lambda: reg strength] SMOOTH[Small smooth weights] SPARSE[Sparse weights, many zero] ROBUST[Robust features] DECAY[Weight decay: 1 minus eta lambda] OF -->|solved by| REG REG -->|technique| L2 REG -->|technique| L1 REG -->|technique| DROP LAM -->|controls tradeoff| L2 LAM -->|controls tradeoff| L1 L2 -->|differentiable, gentle| SMOOTH L1 -->|kink at zero| SPARSE DROP -->|no single pathway| ROBUST L2 -->|gradient yields| DECAY DECAY -->|shrinks weights each step| SMOOTH ``` ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > Regularization ka matlab hai apne ML model ko "discipline" sikhana. Jab ap bohot saare parameters wale model ko limited data pe train karte ho, toh model cheating karne lagta hai—woh training examples ko ratt leta hai, samajhta nahi. Jaise student exam se pehle sirf past papers ke answers yad kar le, concepts na samjhe. Naye questions aye toh fail. > > **L2 regularization** kehta hai: "Bhai, weights ko chhota rakho." Loss function mein ek extra penalty add hota hai: $\frac{\lambda}{2} \sum w_i^2$. Jitne bade weights, utni zyada penalty. Toh model forced hai ki sabko thoda-thoda importance de, kisi ko itna bada na banaye ki noise pe overfit ho jaye. Aircraft drag prediction mein agar Mach number ka weight bohot bada ho gaya, toh tiny sensor error se prediction bilkul galat ho jayegi. L2 use control karta hai—sare weights smooth aur balanced rehte hain. > > **L1 regularization (Lasso)** aur bhi strict hai. Woh kehta hai: "Sirf important features rakho, baki ko zero kar do." Penalty hai $\lambda \sum |w_i|$, absolute value ka. Geometry dekho toh L1 constraint ek diamond shape banata hai jisme corners hain—wahaan weights exactly zero ho jate hain. Aerospace mein agar 50 engine sensors hain lekin sirf 5 actually useful hain, toh L1 baki 45 ko automatically hata dega. Feature selection khud ho gaya, manual effort nahi chahiye. Model fast bhi chalega aur interpretable bhi—engineer ko pata chal jayega ki kaun se sensors critical hain. > > **Dropout** bilkul alag technique hai. Training ke time randomly kuch neurons ko "band" kar dete ho (output zero). Har iteration mein alag neurons drop hote hain. Why? Kyunki isse neuronsek-dusre pe dependent nahi ho pate—"co-adaptation" nahi hota. Har neuron ko sikhna padta hai independently kaam karna. Test time pe sare neurons use karte ho (dropout off), aur model bohot robust ban chuka hota hai. Deep neural networks mein, especially fully-connected layers mein, dropout bohot kaam ata hai. Flight data classification mein jab CNN overfitting kar raha tha (training accuracy 99%, test 78%), dropout ne use force kiya robust features seekhne ko—test accuracy 89% tak aa gayi. > > Teen techniques ko combine bhi kar sakte ho: L ![[audio/5.6.03-Regularization-—-L1-(lasso),-L2-(ridge),-dropout.mp3]]