Worked examples — Regularization — L1 (lasso), L2 (ridge), dropout

This page is the practice arena for the parent topic. We already built the three tools — L2 (ridge), L1 (lasso), dropout — from first principles. Here we run them by hand through every situation you can meet: every sign, the tricky exactly-at-zero case, the two limits $\lambda =0$ and $\lambda \to \infty$, a real aerospace word problem, and an exam-style twist.

Nothing new is assumed. If a symbol shows up, it was defined in the parent note; but we re-state each one the moment we use it, so you can start reading from line one.

Two symbols we lean on constantly:

• $\eta$ (eta) = learning rate, the size of one gradient step. Think "how big a stride."
• $\lambda$ (lambda) = regularization strength, how hard we penalize big weights. Think "how strict the teacher is."

---

The scenario matrix

Every cell below is a kind of situation the topic can throw at you. Each worked example is tagged with the cell(s) it covers, so by the end every cell is filled.

#	Cell (case class)	Tool	Covered by
A	Positive weight, gentle shrink	L2	Ex 1
B	Negative weight, sign handled	L2	Ex 1
C	Limit $\lambda =0$ (no reg)	L2/L1	Ex 2
D	Limit $\lambda \to \infty$ (over-shrink / overshoot)	L2	Ex 2
E	Small weight → crosses zero (soft threshold)	L1	Ex 3
F	Large weight → shrinks but survives, both signs	L1	Ex 3
G	Degenerate input $w=0$ exactly (subgradient)	L1	Ex 4
H	Geometry: why L1 corner = sparsity	L1 vs L2	Ex 5
I	Dropout expectation / scaling both conventions	Dropout	Ex 6
J	Real-world word problem (choose the tool)	all	Ex 7
K	Exam twist (L1+L2 = elastic net, hidden trap)	mixed	Ex 8

---

Example 1 — L2 on a positive and a negative weight (cells A, B)

Forecast: guess before computing — the positive weight should get smaller (toward 0), the negative weight should get less negative (also toward 0). Regularization always pulls toward zero. Hold that prediction.

The recipe (from the parent's L2 formula): $w^{\text{new}}=(1-\eta \lambda )w-\eta \frac{\partial {\mathcal{L}}_{\text{data}}}{\partial w}$

Step 1 — compute the shrink factor $(1-\eta \lambda )$. $1-\eta \lambda =1-(0.1)(0.5)=0.95$ Why this step? This single number is the whole soul of L2: before we even look at the data, every weight is scaled to $95\%$ of itself. That is "weight decay."

Step 2 — apply it to $w_a=+4.0$. $0.95\times 4.0=3.8,3.8-(0.1)(1.0)=3.7$ So $w_a^{\text{new}}=3.7$. Why this step? The shrink ($4.0\to 3.8$) is proportional — a big weight loses a big chunk. Then the data gradient nudges it down a touch more.

Step 3 — apply it to $w_b=-2.0$. $0.95\times (-2.0)=-1.9,-1.9-(0.1)(-0.5)=-1.9+0.05=-1.85$ So $w_b^{\text{new}}=-1.85$. Why this step? Notice the sign is handled automatically — the shrink factor multiplies whatever sign the weight has, so $-2.0$ becomes $-1.9$ (closer to zero, not further). No special case needed for negatives in L2.

Verify: Both moved toward zero: $|4.0|\to 3.7$ and $|-2.0|\to 1.85$. Matches the forecast. The bigger weight decayed by $0.3$, the smaller by $\approx 0.15$ — proportional, as promised. Units: weights are dimensionless model coefficients, so all terms are dimensionless. ✓

---

Example 2 — the two L2 limits (cells C, D)

Forecast: With $\lambda =0$ nothing should move. As $\lambda$ grows the weight is dragged harder toward zero. But is there a $\lambda$ so big it overshoots? Guess yes — and think about what would signal that.

Since the data gradient is $0$, the update is purely $w^{\text{new}}=(1-\eta \lambda )w$.

Step 1 — $\lambda =0$ (cell C). $1-\eta \lambda =1-0=1\Rightarrow w^{\text{new}}=1\times 10=10$ Why this step? With no regularizer the model is untouched — this is the baseline the parent warned overfits.

Step 2 — $\lambda =2$. $1-(0.1)(2)=0.8\Rightarrow w^{\text{new}}=8$ Why this step? A healthy shrink: $80\%$ of the weight survives each step, so over many steps it decays smoothly toward $0$.

Step 3 — $\lambda =20$ (cell D, the danger zone). $1-(0.1)(20)=1-2=-1\Rightarrow w^{\text{new}}=-1\times 10=-10$ Why this step? The shrink factor went negative. The weight didn't shrink — it flipped sign and stayed size $10$! With an even larger $\lambda$ the factor $<-1$ and the weight explodes. This is the practical meaning of "$\lambda \to \infty$": the update becomes unstable unless $\eta \lambda <1$.

Verify: The stability rule is $|1-\eta \lambda |<1$, i.e. $0<\eta \lambda <2$. Check: $\lambda =0\Rightarrow 0$ (edge, no motion), $\lambda =2\Rightarrow 0.2$ ✓ inside, $\lambda =20\Rightarrow 2.0$ exactly on the unstable edge, giving the sign flip we saw. The parent's claim "$\lambda \to \infty$ weights shrink to zero" is the ideal fixed point — but only reachable if $\eta$ is small enough to keep $\eta \lambda <2$. ✓

---

Example 3 — L1 soft-thresholding, small vs large, both signs (cells E, F)

Forecast: L1 subtracts a fixed amount $\eta \lambda =0.05$ toward zero. Guess which weights survive: a weight smaller than $0.05$ can't survive; one much larger will just get $0.05$ shaved off.

The parent's soft-threshold formula: $w^{\text{new}}=\mathrm{sign}(w)\max (0,|w|-\eta \lambda )$

Step 1 — $w_1=0.05$ (cell E, exactly at the threshold). $|w_1|-\eta \lambda =0.05-0.05=0,\max (0,0)=0\Rightarrow w_1^{\text{new}}=0$ Why this step? Its magnitude equals the step, so it lands exactly on zero and is selected out. This is L1's automatic feature selection in action.

Step 2 — $w_2=-0.03$ (cell E, negative, below threshold). $|w_2|-\eta \lambda =0.03-0.05=-0.02,\max (0,-0.02)=0\Rightarrow w_2^{\text{new}}=0$ Why this step? Without the $\max (0,\cdot )$ guard the weight would flip to $-\mathrm{sign}\times$positive — a bogus overshoot. The $\max$ clamps it to exactly zero. The $\mathrm{sign}$ correctly handled the negative input.

Step 3 — $w_3=1.2$ (cell F, large, survives). $|w_3|-\eta \lambda =1.2-0.05=1.15,\mathrm{sign}(1.2)\times 1.15=+1.15\Rightarrow w_3^{\text{new}}=1.15$ Why this step? A big informative weight just loses the fixed $0.05$ — it survives and keeps doing its job. Contrast with L2, which would have taken a proportional $5\%$ ($=0.06$) off it.

Verify: Two of three weights hit exactly zero (sparsity!), the large one shrank by exactly $\eta \lambda =0.05$. Sanity: sum of absolute weights fell from $0.05+0.03+1.2=1.28$ to $0+0+1.15=1.15$, a drop of $0.13$, and $0.13\le 3\times 0.05=0.15$ (can't drop more than the total available step). ✓

---

Example 4 — the degenerate case $w=0$ exactly (cell G)

Forecast: The parent noted $\frac{\partial |w|}{\partial w}$ is undefined at $w=0$ (the diamond has a sharp corner there). So what do we plug in? Guess: the penalty should "resist" the data gradient rather than push a definite direction.

Step 1 — recall the subgradient at zero. At $w=0$ the slope of $|w|$ isn't a single number; it's any value in the interval $[-1,+1]$. This set is called the subgradient. Why this step? Because the corner has many tangent lines, not one — we must reason with the whole interval, not a point.

Step 2 — check whether the data gradient can escape the penalty. The weight leaves zero only if the data gradient is strong enough to beat the maximum penalty pull $\lambda =0.5$. Here $\left|\frac{\partial {\mathcal{L}}_{\text{data}}}{\partial w}\right|=0.3<\lambda =0.5$. Why this step? Inside the interval $[-\lambda ,\lambda ]=[-0.5,0.5]$ the penalty can exactly cancel the data gradient by choosing subgradient $g^{\ast }=-0.3/0.5\cdot$… — more simply, since $0.3<0.5$, some subgradient value makes the total gradient zero.

Step 3 — conclude the weight stays put. $w^{\text{new}}=0$ Why this step? A weight already zeroed by L1 stays zeroed unless its data signal exceeds $\lambda$. This is exactly why L1 gives stable sparsity — features don't flicker back on for weak signals.

Verify: Threshold check: needs $|0.3|>0.5$ to activate; false, so weight stays $0$. If instead the data gradient were $0.8>0.5$, one step would give $w^{\text{new}}=-\eta (0.8-\lambda \cdot \mathrm{sign})=-0.1(0.8-0.5)=-0.03$, i.e. the weight would finally leave zero. ✓

---

Example 5 — geometry: why the L1 diamond makes zeros (cell H)

Forecast: Picture the ellipse growing outward until it first touches the budget region. Where does it touch a circle vs a diamond? Guess the diamond gets touched at a pointy corner.

Step 1 — read the L2 (circle) side, left panel. The budget $w_1^2+w_2^2\le C$ is the round region. The expanding ellipse (data loss) touches it on a smooth arc, at a point where both $w_1$ and $w_2$ are nonzero. Why this step? A smooth boundary has no preferred direction, so the tangency lands wherever the ellipse leans — generically off-axis. Result: all weights small, none exactly zero.

Step 2 — read the L1 (diamond) side, right panel. The budget $|w_1|+|w_2|\le C$ is the diamond. Its corners lie on the axes (where one coordinate is zero). The ellipse, expanding, very often first bumps a corner. Why this step? Corners stick out toward the ellipse and cover a whole range of ellipse orientations, so tangency at a corner is likely. At that corner one coordinate is exactly zero → sparsity.

Step 3 — connect to the update rule. This geometry is the same story as Example 3's soft-threshold: the constant push $\eta \lambda$ is what drives a coordinate all the way onto the axis.

Verify: Corner of the diamond $|w_1|+|w_2|=C$ on the $w_2$-axis is $(0,C)$ — one coordinate exactly $0$. For a circle $w_1^2+w_2^2=C^2$ the only axis points are isolated; a generic tangent line touches off-axis. ✓ (algebraic check in VERIFY)

---

Example 6 — dropout expectation, both scaling conventions (cell I)

Forecast: "Keep probability" is $1-p=0.75$. Guess the expected surviving signal is $0.75\times 6=4.5$ for classic dropout, and that inverted dropout is engineered to give back the full $6$.

Step 1 — classic dropout, training expectation. $\mathbb{E}[\tilde{h}]=(1-p)h=0.75\times 6.0=4.5$ Why this step? On average $75\%$ of forward passes keep the neuron; the mean signal is scaled down to $4.5$.

Step 2 — classic dropout, test time correction. At test we keep the neuron always, so to match training's expected $4.5$ we scale by $(1-p)$: $h^{\text{test}}=(1-p)h=0.75\times 6.0=4.5$ Why this step? Test must see the same average magnitude the next layer was trained on, or every activation would be $1/0.75$ too big.

Step 3 — inverted dropout, training scaling. When kept, divide by $(1-p)$: $\tilde{h}=\frac{h}{1-p}=\frac{6.0}{0.75}=8.0\text{(when kept), }0\text{ (when dropped)}$ $\mathbb{E}[\tilde{h}]=(1-p)\cdot \frac{h}{1-p}+p\cdot 0=h=6.0$ Why this step? By pre-inflating during training, the expectation is already the full $h$, so test time needs no scaling — which is why modern frameworks use it.

Verify: Classic: train mean $4.5$, test output $4.5$ — matched ✓. Inverted: train mean $=6.0=h$, test output $=6.0$ — matched ✓. Both conventions give identical test behaviour ($4.5$ effective vs $6.0$ full is a re-parameterization; the ratio to downstream weights is consistent). ✓

---

Example 7 — real-world: pick the right tool (cell J)

Forecast: Sparsity + interpretability screams one tool; overfitting in the deep net screams another. Guess: L1 on the sensor-selection linear layer, dropout inside the CNN.

Step 1 — sensor selection → L1. Why? From Ex 3, L1 zeroes weak weights exactly, so the 45 useless sensors get coefficient $0$ and drop out of the model. This gives the interpretable "which 5 sensors" answer engineers asked for. Ties to Feature Engineering.

Step 2 — smooth stable coefficients on the survivors → optionally L2. Why? Among the surviving 5 sensors we don't want any single coefficient to blow up (Ex 1's proportional decay), so a light L2 keeps them well-conditioned.

Step 3 — deep CNN overfitting → dropout. Why? From Ex 6, dropout breaks co-adaptation so the CNN can't memorize training-flight noise. It behaves like averaging many sub-networks — related to Ensemble Methods.

Step 4 — pick $\lambda$ / $p$ honestly. Why? Neither strength is known a priori; sweep them with Cross-Validation and watch the gap between train and test loss (Overfitting Detection).

Verify: Sanity on counts: L1 with a moderate $\lambda$ that keeps 5 of 50 features means $\ge 45$ zeros — consistent with the parent's Example 2 vector having 45 zeros. Dropout $p=0.5$ on FC layers matches the parent's fault-detection CNN. ✓

---

Example 8 — exam twist: elastic net trap (cell K)

Forecast: Two forces at once — proportional L2 shrink and constant L1 push. Guess: apply the L2 factor first, then the L1 soft-threshold on the result.

The combined update (data gradient $0$): $w^{\text{new}}=\mathrm{sign}(w)\max (0,|(1-\eta {\lambda }_2)w|-\eta {\lambda }_1)$

Step 1 — apply the L2 shrink factor. $1-\eta {\lambda }_2=1-(0.1)(2)=0.8,0.8\times 0.5=0.4$ Why this step? L2 is proportional, so it acts multiplicatively first, giving $0.4$.

Step 2 — apply the L1 soft-threshold to that result. $\eta {\lambda }_1=(0.1)(1)=0.1,\max (0,0.4-0.1)=0.3,\mathrm{sign}(0.5)\times 0.3=0.3$ Why this step? Now subtract the constant L1 step. Both pulls toward zero, but neither alone reached zero, so the weight survives at $0.3$.

Step 3 — spot the trap. If you (wrongly) applied L1 first: $\max (0,0.5-0.1)=0.4$, then $\times 0.8=0.32$. Different answer! The convention is L2 shrink then L1 threshold, giving $0.3$. Why this step? Because the L1 $\max (0,\cdot )$ is nonlinear, order matters — the classic exam gotcha.

Verify: Correct order gives $0.30$; wrong order gives $0.32$; they differ, confirming order-sensitivity. Both pull the weight down from $0.5$ (elastic net = "L1 sparsity flavor + L2 stability"). ✓

---

Recall Self-test (reveal after answering)

L2 step on $w=4$, data grad $1$, $\eta =0.1$, $\lambda =0.5$ gives what? ::: $3.7$ L1 step zeroes $w=0.05$ when $\eta \lambda$ equals what value? ::: $0.05$ For L2 stability, $\eta \lambda$ must stay below what? ::: $2$ Inverted dropout with $p=0.25$: a kept neuron of value $6$ becomes? ::: $8.0$ Elastic-net order (which penalty first)? ::: L2 shrink, then L1 threshold

Related deep-dive links: Gradient Descent Variants (how $\eta$ interacts with these updates), Neural Network Architectures (where dropout layers sit), Bayesian Inference (L2 = Gaussian prior, L1 = Laplace prior — the probabilistic story).