Exercises — L1 - L2 weight decay in deep nets

$R_a$ is L1 (sum of absolute values → the "lasso"/diamond). $R_b$ is L2 (sum of squares, times $\frac{1}{2}$ → "weight decay"/ridge, circle). The $\frac{1}{2}$ is a convenience so its derivative $\frac{d}{dw}(\frac{1}{2}{w}^2)=w$ comes out clean.

It is the shrink factor. For typical small positive $\eta$ and $\lambda$, the product $\eta \lambda$ is a small positive number, so $(1-\eta \lambda )$ is just below 1. Multiplying by a number below 1 pulls $w$ toward 0 every step — that is the "decay."

Shrink factor $=1-\eta \lambda =1-0.1\cdot 0.5=0.95$. With zero data gradient each step is just $w\leftarrow 0.95w$:

• Step 1: $0.95\cdot 4=3.8$
• Step 2: $0.95\cdot 3.8=3.61$
• Step 3: $0.95\cdot 3.61=3.4295$

Answer: $w_3=3.4295$. This is geometric decay $w_n=4\cdot 0.95^n$ — the weight leaks toward 0 with no data holding it up.

Equilibrium means ${\nabla }_{w}J+\lambda w^{\star }=0$, so $w^{\star }=-\frac{{\nabla }_{w}J}{\lambda }=-\frac{-3}{0.5}=6.$ Why? At $w^{\star }$ the upward data push $-\eta {\nabla }_{w}J=+0.3$ exactly cancels the downward decay $-\eta \lambda w=-0.1\cdot 0.5\cdot 6=-0.3$. Answer: $w^{\star }=6$.

$\eta \lambda =0.1\cdot 1=0.1$. Then $\max (|0.05|-0.1,0)=\max (-0.05,0)=0$. So $w\leftarrow \text{sign}(0.05)\cdot 0=0$. Answer: $w=0$ exactly. The constant toll $0.1$ is bigger than the tiny weight $0.05$, so it is snapped to zero and stays there — sparsity.

L2 (A): shrink factor $0.9$, so $w_n=0.9^n$: $1.0\to 0.9\to 0.81\to 0.729$. Multiplicative — it never hits exactly 0.

L1 (B): subtract $\eta \lambda =0.1$ each step (soft-threshold), so $w_n=\max (1.0-0.1n,0)$: $1.0\to 0.9\to 0.8\to 0.7$. Linear — it will hit exactly 0 at step 10.

Early on they look similar, but see the figure: L2's curve flattens and hugs the axis forever, while L1 marches down a straight line and lands on 0. L1 reaches exact 0; L2 never does.

$w^{\star }=-{\nabla }_{w}J/\lambda$ is inversely proportional to $\lambda$. Doubling $\lambda$ makes $w^{\star }$ half as large — every weight shrinks proportionally. But a weight only reaches 0 if its data gradient ${\nabla }_{w}J$ is 0; otherwise $w^{\star }$ is some nonzero fraction, never exactly 0. So L2 gives uniformly smaller weights, not zeros. Sparsity needs the constant-force L1 push that can overpower a small weight regardless of $\lambda$-proportionality.

Survival test: compare $|{\nabla }_{w}J|=0.2$ against $\lambda =0.5$. Since $\lambda =0.5>0.2=|{\nabla }_{w}J|$, the constant L1 toll beats the data's pull — the weight is driven to 0 (dies).

Contrast: under L2 the same weight settles at $0.4$ (survives, just small). This is the punchline: with identical $\lambda$, L2 keeps the weight (shrunk), L1 kills it. L1 does automatic feature selection — a feature whose gradient can't out-muscle the toll gets removed entirely. See Lasso and Ridge Regression.

In Adam the gradient (including the L2 term $\lambda w$) is divided by each parameter's own $\sqrt{v_i}+ϵ$. So the effective decay on weight $i$ is $\lambda /(\sqrt{v_i}+ϵ)$ — not the clean constant $\lambda$. Parameters with large gradient history $v_i$ get their decay shrunk; parameters with small $v_i$ get it amplified. The intended uniform shrink is scrambled.

AdamW fixes this by decoupling: it applies the adaptive step to the data gradient only, then adds the decay $-\eta \lambda w$ as a separate, un-scaled term: $w\leftarrow w-\eta \frac{\hat{m}}{\sqrt{\hat{v}}+ϵ}-\eta \lambda w.$ Now every weight decays by the same factor $(1-\eta \lambda )$ regardless of $v_i$. That's why AdamW ≠ Adam + L2. See Adam and AdamW Optimizers.

$S_{\tau }(w)=\text{sign}(w)\max (|w|-\tau ,0).$ With $\tau =0.1$:

• $w=0.5$: $\text{sign}(+)\cdot \max (0.5-0.1,0)=+0.4$
• $w=0.05$: $\max (0.05-0.1,0)=0\Rightarrow 0$
• $w=-0.3$: $\text{sign}(-)\cdot \max (0.3-0.1,0)=-0.2$
• $w=0$: $0$

Geometrically: $S_{\tau }$ pulls every value toward zero by a fixed amount $\tau$, and anything already within $\tau$ of zero is flattened onto 0 — creating a "dead zone" $[-\tau ,\tau ]$ that maps to exactly 0. That dead zone is the source of sparsity; L2's shrink has no dead zone, only a slope through the origin.

We need $(1-\eta \lambda {)}^n=0.5$. Here $1-\eta \lambda =1-0.02=0.98$. Take $\ln$ of both sides (the tool that turns the exponent $n$ into a multiplier we can solve for): $n\ln (0.98)=\ln (0.5)\Rightarrow n=\frac{\ln 0.5}{\ln 0.98}=\frac{-0.6931}{-0.02020}\approx \mathrm{34.3.}$ Answer: about $34$ steps to halve. Larger $\eta \lambda$ ⇒ faster decay ⇒ shorter half-life. This is exactly radioactive-decay maths applied to weights — connect to Gradient Descent.

The MAP objective adds $-\log (\text{prior})$ to the loss. The Laplace prior $\propto e^{-\lambda |w|}$ has a sharp corner (kink) at $w=0$ — its negative-log is $\lambda |w|$, whose slope jumps from $-\lambda$ to $+\lambda$ across 0, so 0 is a strong attractor the optimizer can rest exactly on. The Gaussian prior $\propto e^{-\lambda w^2/2}$ is smooth and flat at $w=0$ (zero slope there), so there's no kink pulling the weight onto 0 — it only gently squeezes. Sharp corner ⇒ exact zeros (L1); smooth bump ⇒ small-but-nonzero (L2). See Overfitting and Generalization for why sparsity aids generalization.

L2 = multiply-shrink, never zero, half-life set by $\eta \lambda$ · L1 = subtract-fixed-toll, exact zeros via soft-threshold dead zone · equilibrium $w^{\star }=-\nabla J/\lambda$ · Adam breaks clean decay, AdamW restores it · corner-prior (Laplace) ⇒ sparse, smooth-prior (Gaussian) ⇒ dense.