Visual walkthrough — L1 - L2 weight decay in deep nets

---

Step 1 — What is a "weight" and what does the loss look like?

WHAT. A neural network multiplies its inputs by numbers called weights. Think of a single weight $w$ — just one dial the network can turn. The loss $J(w)$ is a score that tells us how badly the network fits the training data: low = good, high = bad. We want to turn the dial $w$ to the value that makes $J$ smallest.

WHY a picture of $J$ vs $w$. Because "training" is rolling a ball downhill on this curve. If you can see the hill, you can see everything the optimizer does.

PICTURE. The cyan curve is $J(w)$, a bowl. The lowest point (amber dot) is the weight the data alone wants — call it $w_{\text{data}}$. The slope of the curve at any point is written ${\nabla }_{w}J$ (read: "gradient of $J$"); it just means how steep the hill is and which way is uphill.

---

Step 2 — Rolling downhill: plain gradient descent

WHAT. Gradient Descent says: to go down, step opposite the slope. The rule is $w\leftarrow w-\underset{\text{step size}}{\underbrace{\eta }}\underset{\text{uphill slope}}{\underbrace{{\nabla }_{w}J}}$

Term by term: ${\nabla }_{w}J$ points uphill, the minus sign flips it to point downhill, and $\eta$ (the learning rate, a small positive number like $0.1$) sets how big a stride we take.

WHY. This is our baseline. Every later formula is this rule plus one extra force. To see what regularization adds, we must first see what it adds to.

PICTURE. The ball starts high on the bowl and takes arrows down the slope, each arrow length $\eta |{\nabla }_{w}J|$, until it rests at the amber minimum $w_{\text{data}}$.

---

Step 3 — Adding a "laziness tax": the L2 penalty

WHAT. We now add a second cost that has nothing to do with the data — it charges us for making $w$ big. The L2 penalty is $R(w)=\frac{1}{2}{w}^2$ and the total objective becomes $\tilde{J}(w)=\underset{\text{fit the data}}{\underbrace{J(w)}}+\underset{\text{tax rate}}{\underbrace{\lambda }}\underset{\text{price of size}}{\underbrace{\frac{1}{2}{w}^2}}$

Term by term: $\frac{1}{2}{w}^2$ is a parabola — cheap near $w=0$, expensive far away. $\lambda$ (lambda, $\ge 0$) is the regularization strength: how heavily we tax weight size.

WHY a parabola and why the $\frac{1}{2}$. We want a penalty that is smallest at $0$ and grows smoothly. $w^2$ does exactly that. The $\frac{1}{2}$ is pure convenience: its derivative is $\frac{1}{2}\cdot 2w=w$, a clean result with no stray factor. (See Maximum a Posteriori (MAP) Estimation for why a parabola specifically — it is a Gaussian prior in disguise.)

PICTURE. Cyan = the data bowl $J$. Amber = the parabola $\frac{\lambda }{2}{w}^2$ centred at $0$. White = their sum $\tilde{J}$. Notice the white curve's minimum has slid left of the data minimum, toward zero.

---

Step 4 — The slope of the tax is a spring pulling to zero

WHAT. To do gradient descent on $\tilde{J}$ we need its slope. Slopes add, so ${\nabla }_w\tilde{J}=\underset{\text{data force}}{\underbrace{{\nabla }_{w}J}}+\underset{\text{penalty force}}{\underbrace{\lambda w}}$ because the derivative of $\frac{\lambda }{2}{w}^2$ is exactly $\lambda w$.

WHY this matters. Look at the penalty force $\lambda w$: it is proportional to $w$ and points back toward $0$ (if $w>0$ the force is positive, and we subtract it → pushed left; if $w<0$, pushed right). That is precisely a spring anchored at the origin — the further out you are, the harder it pulls you in.

PICTURE. At three different weights, amber arrows show the spring force $\lambda w$ — long when $w$ is far out, short near $0$, vanishing at $0$. This "shrinks as you approach" behaviour is the whole reason L2 never quite reaches exactly zero.

---

Step 5 — Fold the spring into the update: "weight decay" appears

WHAT. Put the new slope into the gradient-descent rule from Step 2: $w\leftarrow w-\eta ({\nabla }_{w}J+\lambda w)$ Now group the two $w$ terms: $w\leftarrow \underset{\text{shrink factor}<1}{\underbrace{(1-\eta \lambda )}}w-\eta {\nabla }_{w}J$

Term by term: multiply the current weight by $(1-\eta \lambda )$ — a number just below 1 — then take the ordinary data step. Since $\eta \lambda$ is small and positive, $(1-\eta \lambda )$ is like $0.95$: each round the weight shrinks 5% before the data even speaks.

WHY the name. Before any data force acts, $w$ is multiplied down toward $0$. It "decays" — hence weight decay. A weight survives only if the data gradient keeps re-supplying it.

PICTURE. One update split into two arrows: a short amber arrow pulling $w$ inward by the factor $(1-\eta \lambda )$, then a cyan data arrow. The final white dot lands where both agreed.

---

Step 6 — Where does the weight finally rest? (equilibrium)

WHAT. The dance stops when an update leaves $w$ unchanged, i.e. the bracket in Step 5 is zero: ${\nabla }_{w}J+\lambda w=0⟹w^{\star }=-\frac{1}{\lambda }{\nabla }_{w}J$

Term by term: $w^{\star }$ is the resting weight; ${\nabla }_{w}J$ is the data's pull; dividing by $\lambda$ says stronger tax ⇒ smaller resting weight.

WHY. This is the "tug of war" settling point: the data force and the spring force exactly cancel. It quantifies the slogan "the data must earn every unit of weight."

PICTURE. Two opposing arrows of equal length at $w^{\star }$: cyan data pull outward, amber spring pull inward, balanced. Below, three curves for growing $\lambda$ show $w^{\star }$ sliding toward $0$.

---

Step 7 — The L1 twist: a constant force, not a spring

WHAT. Swap the penalty from $\frac{1}{2}{w}^2$ to $|w|$ (the L1 penalty, see Lasso and Ridge Regression). Its slope is ${\nabla }_w|w|=\mathrm{sign}(w)=\{\begin{array}{ll}+1& w>0\\ -1& w<0\end{array}$ so the penalty force is $\lambda \mathrm{sign}(w)$ — a fixed size, same whether $w$ is huge or tiny.

WHY it changes everything. L2's spring $\lambda w$ faded out near zero (Step 4) — so it could never finish the job. L1's push stays full strength right up to zero, so it drives small weights all the way to exactly $0$. That is sparsity.

PICTURE. Side by side: L2 arrows shrinking as $w\to 0$ (never arriving), L1 arrows all the same length — the last one overshoots past $0$, so the weight is clamped exactly to $0$ by the soft-threshold rule $w\leftarrow \mathrm{sign}(w)\max (|w|-\eta \lambda ,0).$

---

Step 8 — The degenerate cases (never leave the reader stranded)

WHAT / WHY / PICTURE, all at once:

• $\lambda =0$. No tax. Shrink factor $(1-\eta \lambda )=1$, spring force $0$ — we recover plain gradient descent (Step 2). Weights rest at $w_{\text{data}}$.
• $\lambda$ too large so that $\eta \lambda \ge 1$. Then $(1-\eta \lambda )\le 0$: the shrink factor flips the weight's sign every step or blows it up — training oscillates or diverges. Keep $\eta \lambda <1$.
• $w=0$ exactly (L2). Spring force $\lambda \cdot 0=0$ — nothing pushes; a weight already at $0$ stays unless the data revives it.
• $w=0$ exactly (L1). $\mathrm{sign}(0)$ is undefined — but the soft-threshold rule handles it: $\max (0-\eta \lambda ,0)=0$, so $0$ is a stable resting point. This is exactly why L1 parks weights there.
• $\lambda \to \infty$. From Step 6, $w^{\star }=-{\nabla }_{w}J/\lambda \to 0$: every weight crushed to zero → the network underfits (high bias, see Bias-Variance Tradeoff).

The amber danger-zone panel shows the good regime $0<\eta \lambda <1$ versus the divergent regime $\eta \lambda \ge 1$.

---

The one-picture summary

Everything on one canvas: the data bowl, the penalty (spring for L2 / constant toll for L1), the combined force, the shrink-then-step update, and the resting weight $w^{\star }=-{\nabla }_{w}J/\lambda$ pulled in from the data optimum.

Recall Feynman: retell the whole walkthrough in plain words

Picture a ball on a valley — the valley floor is where the data is happy (Step 1). Normally the ball just rolls down to the floor (Step 2). Now we hang the ball on a rubber band tied to the origin (Step 3). The rubber band pulls harder the further out the ball is (Step 4). So each nudge the ball takes is: *first_ get tugged inward a little, then roll toward the data floor (Step 5). It settles wherever the tug and the roll cancel — and the harder we tune the rubber band ($\lambda$), the closer to zero it parks (Step 6). If instead of a rubber band we tie on a constant weight that pulls the same no matter what (that's L1), it can drag the ball all the way onto zero and pin it there (Step 7). Just don't make the rubber band so strong that it flings the ball out of the valley — keep $\eta \lambda <1$ (Step 8). That "shrink, then step" is the whole idea of weight decay.

Related: adaptive optimizers change this story — see Adam and AdamW Optimizers. A different way to force simplicity is Dropout.