Regularization — L1 (lasso), L2 (ridge), dropout: Exercises

This page is a graded workout for Regularization — L1, L2, dropout. Every symbol used here was built in the parent note; if you meet one you forgot, jump back. Work each problem before opening its solution — the whole point is to catch your own gaps.

Quick symbol refresher, so line one is followable:

Recall The six symbols you need

$w$ (or $w_{i}$ ) ::: a weight — one tunable number the model multiplies a feature by.
$η$ (eta) ::: the learning rate — how big a step gradient descent takes.
$λ$ (lambda) ::: regularization strength — how hard we punish big weights.
$\nabla L_{data}$ ::: the data gradient — the slope of the plain (unpenalized) loss.
$p$ ::: dropout probability — chance a neuron is switched off during training.
$m$ ::: the dropout mask — a coin-flip variable, $m \sim Bernoulli (1 - p)$ , that is $1$ (keep the neuron) with probability $1 - p$ and $0$ (drop it) with probability $p$ .

The four update rules everything below rests on. Read the why beside each — the shape of the penalty is the rule:

Formula The rules you will reuse (with the "why" attached)

L2 (ridge) step: $w \leftarrow (1 - η λ) w - η \nabla L_{data}$ Why the multiplicative factor? The L2 penalty is $\frac{λ}{2} w^{2}$ ; its slope is $λ w$ — proportional to $w$ . Descending that slope subtracts $η λ w$ , i.e. removes a fixed fraction $η λ$ of the weight each step, leaving $(1 - η λ) w$ . A bowl-shaped penalty pushes hardest on big weights and gently on small ones → smooth shrinkage. Stability note: this only shrinks cleanly when $0 < η λ < 1$ ; at $η λ = 1$ the factor is $0$ (weight collapses in one step) and at $η λ \geq 2$ it flips sign / diverges — see Exercise 5.2.

L1 (lasso) step: $w \leftarrow w - η (\nabla L_{data} + λ sign (w))$ Why a constant, not a fraction? The L1 penalty is $λ ∣ w ∣$ ; its slope is $λ sign (w)$ — a constant $\pm λ$ whose size does not depend on $w$ . So every step subtracts the same amount $η λ$ toward zero. A V-shaped penalty has the same steepness everywhere, so even tiny weights feel a full-strength push → they cross zero and stay there (sparsity).

L1 soft-threshold (pure penalty, no data gradient): $w^{new} = sign (w) max (0, ∣ w ∣ - η λ)$ Why the max? Subtracting a constant $η λ$ from a small $∣ w ∣$ would overshoot past zero and flip its sign — which would mean the penalty grew the weight. The $max (0, \cdot)$ clamps at zero so the weight can only shrink, never reverse.

Inverted dropout (train): $\tilde{h} = m \cdot h / (1 - p), m \sim Bernoulli (1 - p);$ test: $h^{test} = h$ Why divide by $(1 - p)$ ? A dropped neuron contributes $0$ , so on average the layer's total signal would shrink to a fraction $(1 - p)$ of its true value — the next layer would see a systematically weaker input during training than at test. Dividing the kept neurons by $(1 - p)$ restores the expectation: $E [\tilde{h}] = (1 - p) \cdot \frac{h}{1 - p} + p \cdot 0 = h$ . Because the training-time expectation already equals the raw $h$ , test time needs no rescaling at all.

Level 1 — Recognition

Exercise 1.1 (L1 · Recognition)

For each phrase, name the technique (L1 / L2 / dropout): (a) "penalizes the squared magnitude of weights" (b) "randomly switches neurons off during training" (c) "drives many weights to exactly zero" (d) "also called weight decay"

Recall Solution

(a) L2 (ridge) — the squared term $\sum w_{i}^{2}$ . (b) dropout — Bernoulli masking of activations. (c) L1 (lasso) — the diamond corners land on zeroed coordinates. (d) L2 (ridge) — the $(1 - η λ)$ factor shrinks weights every step, i.e. decay.

Exercise 1.2 (L1 · Recognition)

A network is at test time. Under standard inverted dropout, what do you do to the activations?

Recall Solution

Nothing. Test time uses the full network with no mask and no scaling, because the $1/ (1 - p)$ scaling was already applied during training. So $h^{test} = h$ .

Level 2 — Application

Exercise 2.1 (L2 · Application)

Given $η = 0.1$ , $λ = 0.5$ , a weight $w = 4$ , and data gradient $\nabla L_{data} = 2$ . Compute one L2 update step.

Recall Solution

Use $w \leftarrow (1 - η λ) w - η \nabla L_{data}$ .

Decay factor: $1 - η λ = 1 - (0.1) (0.5) = 0.95$ .
$w_{new} = 0.95 \times 4 - 0.1 \times 2 = 3.8 - 0.2 = 3.6$ . The weight shrank both from decay ( $4 \to 3.8$ ) and from the data gradient ( $- 0.2$ ).

Exercise 2.2 (L2 · Application)

Same $η = 0.1$ , $λ = 0.5$ , but now turn off the data gradient ( $\nabla L_{data} = 0$ ). Starting at $w = 4$ , how many steps until $∣ w ∣ < 1$ ? (This shows pure decay is geometric.)

Recall Solution

With no data gradient, each step multiplies by $0.95$ : $w_{t} = 4 (0.95)^{t}$ . We want $4 (0.95)^{t} < 1 \Rightarrow (0.95)^{t} < 0.25 \Rightarrow t > \frac{ln 0.25}{ln 0.95} = \frac{- 1.386}{- 0.0513} \approx 27.03$ . Because $t$ counts whole steps, a value of $27.03$ means step 27 is not yet enough — we must round up to the next integer: $t = ⌈ 27.03 ⌉ = 28$ steps. (Always take the ceiling when solving "how many discrete steps until a threshold" — a fractional step doesn't exist.) This is why L2 shrinks but never zeroes a weight — geometric decay approaches zero without reaching it. (Note this holds only because $0 < η λ = 0.05 < 1$ ; if $η λ \geq 1$ the geometric picture breaks — see Exercise 5.2.)

Exercise 2.3 (L1 · Application)

Given $η = 0.1$ , $λ = 2$ , so $η λ = 0.2$ . Apply the soft-threshold rule to each weight (pure penalty step): $w = [0.15, - 0.30, 0.05, - 1.00]$ .

Figure — Regularization — L1 (lasso), L2 (ridge), dropout

Recall Solution

Rule: $w^{new} = sign (w) max (0, ∣ w ∣ - 0.2)$ .

$0.15$ : $∣ w ∣ = 0.15 < 0.2 \Rightarrow max (0, - 0.05) = 0 \Rightarrow 0$ .
$- 0.30$ : $∣ w ∣ = 0.30$ , $0.30 - 0.2 = 0.10$ , sign $- 1 \Rightarrow - 0.10$ .
$0.05$ : $0.05 < 0.2 \Rightarrow 0$ .
$- 1.00$ : $1.00 - 0.2 = 0.80$ , sign $- 1 \Rightarrow - 0.80$ . Result: $[0, - 0.10, 0, - 0.80]$ . Two weights killed exactly, the survivors shrank by the same constant $0.2$ .

Figure s01 (described): the horizontal axis is the incoming weight $w$ , the vertical axis is its new value. A dashed black diagonal is the "do nothing" identity line ( $new = w$ ). The red curve is the soft-threshold output: it is a flat line stuck at zero across the whole band $- 0.2 < w < 0.2$ (shaded red — the "dead zone"), then it splits off parallel to the identity line but shifted toward zero by $0.2$ . Any weight landing inside the red band snaps to exactly zero; anything outside survives but pulled $0.2$ closer to zero.

Level 3 — Analysis

Exercise 3.1 (L3 · Analysis)

Two weights start equal at $w = 0.30$ . One is trained with L2 ( $1 - η λ = 0.9$ per step), the other with L1 (constant $η λ = 0.05$ per step, soft-threshold). Ignore the data gradient. After exactly 6 steps, which is smaller, and which (if any) is zero?

Recall Solution

L2: $w_{6} = 0.30 (0.9)^{6} = 0.30 \times 0.531441 = 0.1594$ — still positive, never exactly zero. L1: subtract $0.05$ each step: after $6$ steps we'd subtract $0.30$ , but it hits zero first.

$0.30 \to 0.25 \to 0.20 \to 0.15 \to 0.10 \to 0.05 \to 0.00$ . That is $6$ subtractions, landing exactly at $0$ on step 6. So L1 is smaller (=0); L2 is $0.1594$ . This is the whole story: L1 sparsifies, L2 only shrinks.

Figure s02 (described): the horizontal axis is the training step $t$ (0 to 6), the vertical axis is the weight value. A black curve (circles) is L2: it bends, each point $0.9 \times$ the last, flattening as it approaches — but never reaching — the zero line. A red line (squares) is L1: a perfectly straight downhill ramp dropping $0.05$ per step, striking the zero axis exactly at $t = 6$ , where a red dot marks the landing. The straight red ramp vs the bending black curve is the entire L1-vs-L2 contrast in one picture.

Exercise 3.2 (L3 · Analysis)

Explain, using the constraint-region picture, why L1 zeros coordinates but L2 does not. Reference the diamond vs circle.

Recall Solution

Regularization = minimize data loss subject to a budget on weights. The budget region is:

L2: $w_{1}^{2} + w_{2}^{2} \leq C$ — a smooth circle (no special points).
L1: $∣ w_{1} ∣ + ∣ w_{2} ∣ \leq C$ — a diamond whose corners sit on the axes (one coordinate $= 0$ ). The data-loss contours are ellipses that expand until they first touch the budget region. A smooth circle is touched at a generic point → both weights nonzero. A diamond is most likely touched at a sharp corner → one weight exactly zero. That corner-catching is geometric sparsity.

Figure s03 (described): centred on the origin are two black budget shapes — a solid circle (L2) and a dashed diamond whose four corners sit exactly on the $w_{1}$ and $w_{2}$ axes (L1). Off to the right sit faint black concentric ellipses, the data-loss contours, growing outward from their own centre. Where the growing ellipse first meets the diamond is a red dot sitting right on the $w_{1}$ -axis, i.e. at $w_{2} = 0$ — a sparse solution. The corner reaches out and "catches" the contour; a smooth circle would have been grazed at an off-axis point with both weights nonzero.

Level 4 — Synthesis

Exercise 4.1 (L4 · Synthesis)

A hidden layer outputs activation $h = 3.0$ . You use inverted dropout with $p = 0.4$ . (a) During training, when the neuron is kept, what value flows forward? (b) Verify the expected training activation equals the raw $h$ . (c) What flows forward at test time?

Recall Solution

Keep probability $= 1 - p = 0.6$ . Inverted rule: kept value $= h / (1 - p)$ . (a) $3.0/0.6 = 5.0$ . (b) $E [\tilde{h}] = (1 - p) \cdot \frac{h}{1 - p} + p \cdot 0 = h = 3.0$ . ✓ Expectation is preserved — that is the entire reason to scale up during training. (c) Test time uses no mask, no scaling: $h^{test} = 3.0$ .

Exercise 4.2 (L4 · Synthesis)

You must predict aircraft drag from 50 candidate features, but you know only a handful are physically relevant and you want the model to tell you which. You also want smooth, stable coefficients on the survivors. Which regularizer(s) do you pick, and why? What would you tune with cross-validation?

Recall Solution

Use L1 (or Elastic Net = L1 + L2):

L1 performs automatic feature selection — zeroing the irrelevant 45-ish sensors, doing feature selection for free and giving interpretable results.
Adding a small L2 term keeps the surviving coefficients stable when features are correlated (pure L1 arbitrarily picks one of two correlated sensors). Tune $λ$ (and the L1/L2 mix) by cross-validation: sweep $λ$ , pick the value minimizing validation error. Confirm you are not overfitting with overfitting detection (train-vs-validation gap).

Level 5 — Mastery

Exercise 5.1 (L5 · Mastery)

A fault-detection network overfits: train accuracy 99%, test 78%. You add dropout $p = 0.5$ on the fully-connected layers and get train 92%, test 89%. (a) By how many points did the generalization gap (train − test) shrink? (b) Training accuracy fell. Explain why that is expected, not a bug. (c) Name one alternative regularizer and one complementary diagnostic from the connections list.

Recall Solution

(a) Before: gap $= 99 - 78 = 21$ pts. After: gap $= 92 - 89 = 3$ pts. Shrinkage $= 21 - 3 = 18$ points. (b) Each training step now runs a random sub-network, so the model can no longer memorize exact examples — training accuracy must drop. We trade a little training fit for a large test gain: the whole objective of regularization. (c) Alternative regularizer: L2 weight decay on the same fully-connected layers (or ensemble methods — dropout is itself an implicit ensemble of $\sim 2^{n}$ sub-networks, so building a real ensemble pushes generalization the same direction). Complementary diagnostic: overfitting detection — plot the train-vs-validation accuracy curves and watch the gap; pair it with cross-validation to actually choose the dropout rate $p$ rather than guessing $0.5$ .

Exercise 5.2 (L5 · Mastery)

Derive, from the plain gradient-descent update, the L2 "weight decay" form, and state precisely the condition on $η λ$ for the decay to be stable (weights don't blow up or oscillate).

Recall Solution

Start from total loss $L_{total} = L_{data} + \frac{λ}{2} \sum_{i} w_{i}^{2}$ (sum over weights only — the bias $b$ is excluded by convention). Its gradient w.r.t. $w_{i}$ : $\frac{\partial L _{total}}{\partial w _{i}} = \frac{\partial L _{data}}{\partial w _{i}} + λ w_{i}$ (the $\frac{1}{2}$ cancels the exponent's $2$ — that's why we put it there). Plug into $w_{i} \leftarrow w_{i} - η \frac{\partial L _{total}}{\partial w _{i}}$ : $w_{i} \leftarrow w_{i} - η (\frac{\partial L _{data}}{\partial w _{i}} + λ w_{i}) = (1 - η λ) w_{i} - η \frac{\partial L _{data}}{\partial w _{i}} .$ The factor $(1 - η λ)$ is weight decay. Stability of the pure-decay part $w_{t + 1} = (1 - η λ) w_{t}$ : geometric with ratio $r = 1 - η λ$ .

Decays monotonically toward 0 when $0 < 1 - η λ < 1 \Rightarrow 0 < η λ < 1$ .
Boundary $η λ = 1$ : $r = 0$ , so the weight collapses to exactly zero in a single step — usable but drastic (all decay, no memory of the old value).
If $1 < η λ < 2$ then $- 1 < r < 0$ : the weight flips sign each step (oscillates while shrinking); if $η λ \geq 2$ then $∣ r ∣ \geq 1$ and it diverges. So keep $η λ < 1$ for clean shrinkage; treat $η λ = 1$ as the knife-edge and anything above it as unsafe.

Recall Self-check summary

L1 vs L2 in one line? ::: L1 subtracts a constant (→ sparsity), L2 multiplies by $(1 - η λ)$ (→ smooth shrinkage, never zero). What is $sign (0)$ and why does it matter? ::: Formally undefined (a subgradient anywhere in $[- 1, 1]$ ); optimizers set it to $0$ so a weight already at zero stays put — which is why soft-thresholding is the safe form. Dropout at test time? ::: Nothing — inverted dropout already scaled during training. Why divide by $(1 - p)$ during training? ::: To preserve the expected activation ( $E [\tilde{h}] = h$ ) so no rescaling is needed at test. L2 stability range? ::: $0 < η λ < 1$ for clean decay; $η λ = 1$ collapses a weight to zero in one step; $η λ \geq 2$ diverges. Do we regularize the bias? ::: No — by convention $b$ is excluded from the penalty since it shifts, not scales, and cannot cause overfitting.