L1 - L2 weight decay in deep nets: Worked examples

This page is a drill-ground for the parent topic. We take the two update rules you already met and run them through every kind of situation they can face: positive and negative weights, zero weights, huge weights, zero data-gradient and nonzero data-gradient, the sparsity snap of L1, the equilibrium of L2, and a limiting case where the step size is dangerously large.

Two rules, kept here so we never look them up mid-example.

A word on each symbol before we use it, so a newcomer is never lost:

$g$ — one number telling us which way the data wants this weight to move and how hard.
$η$ — how big a step we take; a small positive number like $0.1$ .
$λ$ — how much we punish large weights; $0$ means "no punishment."
$sign (u)$ — just the arrow: $+ 1$ if $u > 0$ , $- 1$ if $u < 0$ , $0$ if $u = 0$ .
$max (a, 0)$ — "keep $a$ if it is positive, otherwise clamp to $0$ ." This is the clamp that lets L1 hit exactly zero.

The scenario matrix

Every cell below is a distinct behaviour these rules can produce. The examples that follow are tagged with the cell they cover.

#	Case class	What is special about it	Covered by
A	L2, $g = 0$ , $w > 0$	pure decay, geometric shrink	Ex 1
B	L2, $g \neq = 0$	equilibrium: decay balances data	Ex 2
C	L2, $w < 0$	decay works symmetrically on negatives	Ex 3
D	L1, small $w > 0$ , $g = 0$	snap to exactly $0$ (sparsity)	Ex 4
D'	L1, small $w < 0$ , $g = 0$	snap to $0$ from the negative side	Ex 4b
D''	L1, small $w$ , $g \neq = 0$	still snaps once $∥ u ∥ < η λ$	Ex 4c
E	L1, large $	w	$,$ g\ne 0$
F	L1 vs L2 head-to-head	same weight, different fate	Ex 6
G	Degenerate: $λ = 0$	regularization off, plain GD	Ex 7
H	Limiting: $η λ \geq 1$	factor $\leq 0$ : oscillation / neutral / blow-up	Ex 8
I	Word problem	tuning $λ$ against overfitting	Ex 9
J	Exam twist	Adam ≠ Adam+L2, needs AdamW	Ex 10

We cover every cell with the examples below. Figures accompany the two most geometric ones.

The first figure previews the whole page: it overlays the L2 geometric slide of Example 1 (blue) against the L1 march-to-zero of Example 4 (pink), so you can see before the algebra that L2 curves gently toward $0$ but never lands, while L1 walks in equal steps and hits $0$ exactly. Refer back to it as you read Examples 1 and 4.

Figure — L1 - L2 weight decay in deep nets

Example 1 — Cell A: pure L2 decay, no data

Worked example A weight nobody is pulling on

$η = 0.1$ , $λ = 0.5$ , data gradient $g = 0$ , start $w_{0} = 4$ . Give $w_{1}, w_{2}, w_{3}$ .

Forecast: guess whether the weight halves, drops by a fixed amount, or shrinks by a fixed percentage each step. (The blue curve in the figure above is your answer key.)

Shrink factor $= 1 - η λ = 1 - 0.1 \cdot 0.5 = 0.95$ . Why this step? With $g = 0$ the update collapses to $w \leftarrow (1 - η λ) w$ — nothing but multiplication by $0.95$ .
$w_{1} = 0.95 \cdot 4 = 3.8$ . Why this step? One step is just "current weight $\times$ shrink factor"; there is no data term to add.
$w_{2} = 0.95 \cdot 3.8 = 3.61$ . Why this step? Feed $w_{1}$ back into the same rule — the update is applied afresh each step to the newest weight.
$w_{3} = 0.95 \cdot 3.61 = 3.4295$ . Why this step? Each step multiplies again by the same factor, so this is geometric (percentage) decay, not fixed-amount decay — matching the gently curving blue line in the figure.

Verify: $w_{3} = 4 \cdot 0.9 5^{3} = 4 \cdot 0.857375 = 3.4295$ . ✅ Matches. It never reaches $0$ in finite steps — that is the signature of L2.

Example 2 — Cell B: L2 equilibrium

Worked example Where does the weight settle when data pulls back?

$η = 0.1$ , $λ = 0.5$ , constant data gradient $g = - 3$ . Find the equilibrium weight $w^{⋆}$ and confirm it is stable.

Forecast: will the weight run off to infinity, or park at a finite value?

Equilibrium means the update leaves $w$ unchanged: $w^{⋆} = (1 - η λ) w^{⋆} - η g$ . Why this step? A settled weight is a fixed point of the update map — after a step it is right where it started.
Cancel $w^{⋆}$ : $0 = - η λ w^{⋆} - η g \Rightarrow w^{⋆} = - \frac{g}{λ}$ . Why this step? Divide by $- η$ (nonzero) and solve; note $η$ dropped out — the resting place does not depend on step size.
Plug in: $w^{⋆} = - \frac{- 3}{0.5} = 6$ . Why this step? Here $g = - 3$ is negative, so by the sign rule above it props the weight up; the two minus signs cancel to give a positive resting weight, and $λ = 0.5$ in the denominator doubles it.
Stability: the map's slope is $1 - η λ = 0.95$ , and $∣0.95∣ < 1$ , so any deviation shrinks — it truly settles at $6$ . Why this step? A fixed point is stable when the multiplier's magnitude is below $1$ ; a nudge away from $6$ is multiplied by $0.95$ and fades.

Verify: apply the update at $w = 6$ : $(0.95) (6) - 0.1 (- 3) = 5.7 + 0.3 = 6$ . ✅ Unchanged. Bigger $λ$ would give a smaller $w^{⋆}$ — data must earn each unit of weight.

Example 3 — Cell C: L2 on a negative weight

Example 4 — Cell D: L1 snaps a small positive weight to exactly zero

Worked example The sparsity moment

$η = 0.1$ , $λ = 1$ , $g = 0$ , start $w = 0.05$ . Apply the L1 soft-threshold update.

Forecast: does the weight become slightly negative, or land on exactly $0$ ? (The pink track in the top figure is the shape of the answer.)

$u = w - η g = 0.05 - 0 = 0.05$ . Why this step? With no data gradient, the pre-clamp value $u$ is just the current weight.
Threshold amount $η λ = 0.1 \cdot 1 = 0.1$ . Why this step? The soft-threshold subtracts this fixed distance from $∣ u ∣$ — it is the constant L1 "toll."
$∣ u ∣ - η λ = 0.05 - 0.1 = - 0.05$ , and $max (- 0.05, 0) = 0$ . Why this step? The constant L1 push ( $0.1$ ) is larger than the weight ( $0.05$ ), so the clamp catches it at $0$ instead of letting it overshoot to $- 0.05$ .
$w \leftarrow sign (0.05) \cdot 0 = 0$ . Why this step? Once the trimmed magnitude is $0$ , the sign no longer matters — the weight is exactly zero.

Verify: once $w = 0$ , next step gives $u = 0$ , $max (- 0.1, 0) = 0$ — it stays at $0$ forever. ✅ Exact sparsity, and it is absorbing.

The next figure is the reason this snap happens. It plots the soft-threshold map: input $u$ across, new weight up. The pink band around the origin is the dead zone — every $∣ u ∣ < 0.1$ is flattened onto the line "output $= 0$ ." Example 4's $u = 0.05$ sits inside that band, so it is sent to $0$ ; Example 5's $u = 5.2$ sits far outside it, so it only slides down by the fixed gap; Example 4c's $u = 0.08$ (below) also lands inside the band. Watch how the band is symmetric — that symmetry is exactly what Example 4b relies on.

Example 4b — Cell D': L1 snaps a small negative weight to zero

Worked example Sparsity from the negative side

$η = 0.1$ , $λ = 1$ , $g = 0$ , start $w = - 0.05$ . Apply the L1 soft-threshold update.

Forecast: does a small negative weight snap to $0$ just like the positive one, or does the sign break the symmetry?

$u = w - η g = - 0.05 - 0 = - 0.05$ . Why this step? No data gradient, so $u$ is the current (negative) weight.
$∣ u ∣ = 0.05$ , threshold $η λ = 0.1$ . Why this step? The soft-threshold works on the magnitude $∣ u ∣$ , which is the same $0.05$ as the positive case — that is where the symmetry lives; on the map figure this $u$ sits inside the same pink dead zone, just on the left.
$∣ u ∣ - η λ = 0.05 - 0.1 = - 0.05$ , and $max (- 0.05, 0) = 0$ . Why this step? Same constant toll exceeds the same tiny magnitude, so the clamp fires again.
$w \leftarrow sign (- 0.05) \cdot 0 = (- 1) \cdot 0 = 0$ . Why this step? The sign is $- 1$ , but it multiplies a zero magnitude — so the result is exactly $0$ , not $- 0$ nor a small negative.

Verify: landing point is $0$ , identical to Example 4. ✅ The dead zone in the second figure is symmetric about the origin, so negative weights of small magnitude are zeroed exactly like positive ones.

Example 4c — Cell D'': L1 snaps a small weight to zero even with nonzero data gradient

Worked example Snap survives a data push, as long as it stays small

$η = 0.1$ , $λ = 1$ , data gradient $g = 0.3$ , start $w = 0.11$ . Apply the L1 soft-threshold update.

Forecast: the data gradient is nonzero this time — does the weight escape the dead zone, or still snap to $0$ ?

$u = w - η g = 0.11 - 0.1 \cdot 0.3 = 0.11 - 0.03 = 0.08$ . Why this step? Here $g = 0.3 > 0$ (raising $w$ worsens loss), so by the sign rule the data term $- η g = - 0.03$ pushes $w$ down first, giving the pre-clamp value $u = 0.08$ .
Threshold $η λ = 0.1$ . Why this step? Same fixed L1 toll — it does not care that a data gradient was applied; it acts on whatever $u$ came out.
$∣ u ∣ - η λ = 0.08 - 0.1 = - 0.02$ , and $max (- 0.02, 0) = 0$ . Why this step? The data push shrank $u$ into the dead zone ( $∣ u ∣ = 0.08 < 0.1$ ), so the clamp fires — this is Example 4c's dot inside the pink band on the second figure.
$w \leftarrow sign (0.08) \cdot 0 = 0$ . Why this step? Once inside the dead zone, the outcome is exactly $0$ regardless of the nonzero gradient — sparsity is decided by whether $∣ u ∣ < η λ$ , not by whether $g = 0$ .

Verify: $u = 0.08$ and $∣ u ∣ = 0.08 < 0.1 = η λ$ , so the clamp gives $0$ . ✅ Nonzero-gradient case still snaps — it is the size of $∣ u ∣$ that matters, exactly as the dead-zone figure shows.

Example 5 — Cell E: L1 on a large weight (constant shrink, no snap)

Worked example Big weights just lose a fixed sliver

$η = 0.1$ , $λ = 1$ , data gradient $g = - 2$ , start $w = 5$ .

Forecast: will this large weight snap to $0$ like Example 4, or only get slightly trimmed?

$u = w - η g = 5 - 0.1 \cdot (- 2) = 5 + 0.2 = 5.2$ . Why this step? By the sign rule, $g = - 2$ props the weight up, so the data term $- η g = + 0.2$ nudges $w$ upward before L1 trims it.
Threshold $η λ = 0.1$ . Why this step? Same fixed toll as before — L1's push size never depends on how big the weight is.
$∣ u ∣ - η λ = 5.2 - 0.1 = 5.1 > 0$ , so no clamp: $max (5.1, 0) = 5.1$ . Why this step? The magnitude far exceeds the toll, so we land far outside the pink dead zone of the second figure (at $u = 5.2$ , off to the right) — no snap.
$w \leftarrow sign (5.2) \cdot 5.1 = 5.1$ . Why this step? For large weights L1 removes only the fixed sliver $η λ = 0.1$ — it never snaps them, only the tiny ones die.

Verify: $5.1$ is exactly $5.2 - 0.1$ , a fixed subtraction independent of how big $w$ was. ✅ This is why L1 leaves a few large "survivor" weights while zeroing the crowd of small ones.

Example 6 — Cell F: L1 vs L2 on the same weight

Example 7 — Cell G: degenerate $λ = 0$

Example 8 — Cell H: limiting cases, $η λ \geq 1$ (all three regimes)

Worked example When decay is too aggressive

Study the L2 shrink factor $f = 1 - η λ$ with $g = 0$ , $w_{0} = 1$ , across three settings: (i) $η = 0.5, λ = 3$ ; (ii) $η = 1, λ = 2$ ; (iii) $η = 1, λ = 3$ . Give $w_{1}, w_{2}$ for each and classify the behaviour.

Forecast: guess whether each one decays peacefully, oscillates with steady size, or blows up.

(i) damped oscillation: $f = 1 - 0.5 \cdot 3 = - 0.5$ . $w_{1} = - 0.5$ , $w_{2} = (- 0.5) (- 0.5) = 0.25$ . Why this step? $η λ = 1.5$ gives a negative factor, so the sign flips each step; but $∣ f ∣ = 0.5 < 1$ , so magnitude still shrinks — oscillation that dies out.
(ii) neutral oscillation (the boundary $η λ = 2$ ): $f = 1 - 1 \cdot 2 = - 1$ . $w_{1} = - 1$ , $w_{2} = (- 1) (- 1) = 1$ . Why this step? Exactly at $η λ = 2$ the factor is $- 1$ : the weight flips sign every step but never changes size — it bounces $1, - 1, 1, - 1, \dots$ forever, neither converging nor diverging.
(iii) divergence: $f = 1 - 1 \cdot 3 = - 2$ . $w_{1} = - 2$ , $w_{2} = (- 2) (- 2) = 4$ . Why this step? Now $∣ f ∣ = 2 > 1$ : each step flips sign and doubles magnitude ( $w_{0} = 1 \to w_{1} = - 2 \to w_{2} = 4$ ), so the weight explodes without bound.
Stability rule: the fixed point $w = 0$ is stable iff $∣ f ∣ = ∣1 - η λ ∣ < 1$ , i.e. $0 < η λ < 2$ . The boundary $η λ = 2$ is the neutral knife-edge; beyond it, divergence. Why this step? Convergence of repeated multiplication by $f$ needs $∣ f ∣ < 1$ ; equality gives constant amplitude, and $∣ f ∣ > 1$ grows without bound. Case (i) sits inside the window, (ii) on the boundary, (iii) outside it.

Verify: (i) $w_{2} = (- 0.5)^{2} = 0.25$ ; (ii) $w_{2} = (- 1)^{2} = 1$ (unchanged from $w_{0}$ , confirming neutral); (iii) $w_{2} = (- 2)^{2} = 4$ . ✅ Window $0 < η λ < 2$ ; case (i) inside, case (ii) on the boundary, case (iii) outside.

Example 9 — Cell I: word problem (tuning $λ$ )

Worked example Choosing

$λ$ for a leaky model A net gets $2%$ training error but $18%$ validation error — classic overfitting. You try $λ \in {0, 1 0^{- 4}, 1 0^{- 2}, 1}$ and record validation error ${18%, 9%, 7%, 15%}$ . Which $λ$ do you pick, and what do the endpoints teach?

Forecast: guess whether "biggest $λ$ " wins.

Read the U-shape: error falls, hits a minimum, then rises. Why this step? This is the Bias-Variance Tradeoff made visible — too little regularization = high variance, too much = high bias.
Minimum validation error is $7%$ at $λ = 1 0^{- 2}$ . Pick that. Why this step? We tune $λ$ on validation, not training, and choose the lowest-validation-error point.
Endpoints: $λ = 0$ gives $18%$ (overfit, variance dominates); $λ = 1$ gives $15%$ (underfit, bias dominates). Why this step? Confirms "bigger $λ$ is not always better" — the parent's second mistake, quantified.

Verify: $min (18, 9, 7, 15) = 7$ at index $λ = 1 0^{- 2}$ . ✅ The sweet spot is interior, not at either extreme.

Example 10 — Cell J: exam twist (Adam ≠ Adam+L2)

Worked example Why AdamW exists

In Adam, an L2 penalty adds $λ w$ to the gradient, but Adam divides the gradient by a per-parameter scale $v + ϵ$ . Take a weight with $g = 0$ (only the penalty acts), $λ = 0.1$ , $w = 2$ , learning rate $η = 0.01$ , and adaptive scale $v + ϵ = 0.001$ . Compare the effective decay to what "clean" weight decay ( $η λ w$ ) would do.

Forecast: will the L2-inside-Adam decay be the same size as clean decay?

L2 inside Adam: penalty gradient $= λ w = 0.1 \cdot 2 = 0.2$ ; Adam scales it: effective step $= η \cdot \frac{λ w}{v + ϵ} = 0.01 \cdot \frac{0.2}{0.001} = 2.0$ . Why this step? The small adaptive scale ( $0.001$ ) amplifies the penalty enormously — a weight with tiny gradient variance gets a huge, uncontrolled decay.
Clean (decoupled) decay, as in AdamW: $η λ w = 0.01 \cdot 0.1 \cdot 2 = 0.002$ . Why this step? AdamW applies decay outside the adaptive scaling, so it stays a predictable $(1 - η λ)$ shrink.
Ratio $= 2.0/0.002 = 1000$ . Why this step? The two "decays" differ by a factor of $1000$ here — they are not the same operation.

Verify: $0.01 \cdot 0.2/0.001 = 2.0$ and $0.01 \cdot 0.1 \cdot 2 = 0.002$ ; their ratio is $1000$ . ✅ This is exactly why AdamW decouples decay from the gradient (parent's first mistake).

Recall One-line recap of every cell

A: geometric shrink. B: parks at $- g / λ$ . C: negatives shrink toward $0$ too. D/D': L1 snaps small weights (either sign) to exact $0$ . D'': snap survives a nonzero $g$ if $∣ u ∣ < η λ$ . E: L1 trims big weights by a fixed sliver. F: same weight → L1 zeroes, L2 shrinks. G: $λ = 0$ ⇒ plain GD. H: $η λ < 2$ converges, $η λ = 2$ neutral oscillation, $η λ > 2$ diverges. I: pick $λ$ at validation minimum. J: Adam+L2 ≠ AdamW.

Which cell shows L1 achieving exact sparsity from the negative side?

Cell D' (Example 4b) — small negative weight, magnitude inside the dead zone, clamped to

0

Does L1 still snap a small weight when the data gradient is nonzero?

Yes (Cell D'', Example 4c) — what matters is whether

∣ u ∣ = ∣ w - η g ∣ < η λ

, not whether

g = 0

What is the L2 stability window for the shrink factor, and what happens at the boundary?

0 < η λ < 2

converges;

η λ = 2

gives factor

- 1

and neutral (constant-amplitude) oscillation;

η λ > 2

diverges.

Equilibrium weight formula (L2, constant

g

w^{⋆} = - g / λ

, independent of

η

How does the sign of

g

enter the update?

Via

- η g

g > 0

pushes

w

down,

g < 0

pushes

w

up (descent goes opposite the loss-increasing direction).