Foundations — L1 - L2 weight decay in deep nets

This page assumes you know nothing. Before you can read the parent note L1 / L2 Weight Decay, every symbol it throws at you must first be built from the ground up. We go one symbol at a time, each resting on the last.

1. What is a "weight"? The symbol $w$

The picture. Imagine a knob on a mixing board. Turning it up makes one input signal louder in the final output; turning it down mutes it. Each $w_i$ is one such knob. A deep net has millions of knobs.

Why the topic needs it. The entire note is about controlling the sizes of these numbers. If you don't know $w$ is "the dials we tune", nothing else makes sense.

• The subscript $i$ in $w_i$ just means "the $i$-th knob". $w_3$ = the third knob.
• A scalar is a single number ($w_3=0.7$). A vector is an ordered list of numbers ($w=(0.7,-1.2,0.05)$) — think of it as one point whose coordinates are all the knob positions at once.

2. "How big is the weight vector?" — the norm $\Vert w\Vert$

The penalty measures bigness. To penalize size, we must first say what "size of a vector" means. That measurement is called a norm, written with double bars $\Vert \cdot \Vert$.

The picture. Stand at the origin and put a point at $w=(3,4)$.

• Straight flight (L2): $\sqrt{3^2+4^2}=5$.
• Taxi ride (L1): $|3|+|4|=7$.

Same point, two rulers. The topic uses both rulers because they penalize weights in geometrically different ways — and that difference is the whole L1-vs-L2 story.

• The symbol ${\sum }_i$ ("sigma-sub-i") is shorthand for "add up over every $i$". ${\sum }_i{w}_i^2=w_1^2+w_2^2+\dots$. It is just an addition machine, nothing scarier.
• The bars $|w_i|$ mean absolute value — throw away the minus sign. $|-3|=3$. We use it because a weight of $-5$ is just as "big" as $+5$.

3. What does "good" mean? The loss $J(w)$

The picture. Picture a hilly landscape. The horizontal ground = all possible knob settings $w$. The height at each spot = the loss $J$. Training walks downhill toward a valley.

Why the topic needs it. Weight decay changes this landscape. The parent note writes the new landscape as $\tilde{J}$ (J-with-a-tilde) — the original loss plus a penalty. Everything else is a comparison between $J$ and $\tilde{J}$.

4. Slope of the landscape — the gradient ${\nabla }_{w}J$

To walk downhill you need to know which way is down. That direction is the gradient.

The picture. Standing on a hillside, the gradient is the arrow pointing straight up the steepest slope. Water flows the other way. Each component $\frac{\partial J}{\partial w_i}$ answers: "if I nudge only knob $i$, how fast does the loss change?"

• The symbol $\nabla$ (nabla) is just "collect all the slopes into one arrow".
• $\frac{\partial }{\partial w_i}$ ("partial derivative with respect to $w_i$") means "the slope in the $w_i$ direction, holding all other knobs fixed." A derivative is a rate of change — how much output moves per tiny wiggle of input.

Why we need a derivative here. We want the fastest way down a many-dimensional surface. The derivative is the one tool that answers "which direction changes the loss most, and how fast" — no other quantity gives us a direction to move. See Gradient Descent for the full machinery.

5. How big a step? The learning rate $\eta$

The picture. Down the hill, $\eta$ is your stride length. The update rule $w\leftarrow w-\eta {\nabla }_{w}J$ reads: "move each knob a little bit ($\eta$) in the downhill direction ($-{\nabla }_{w}J$)." The arrow $\leftarrow$ means "replace the left value with the right" — it is an assignment, not an equation.

Why the topic needs it. The famous shrink factor $(1-\eta \lambda )$ that makes "weight decay" decay is built from $\eta$ multiplied by the penalty strength. Without $\eta$ there is no decay.

6. How hard to penalize? The strength $\lambda$

The picture. A seesaw. On one side sits "fit the data" (lower $J$); on the other sits "stay small" (lower $\Vert w\Vert$). $\lambda$ is how much weight you drop onto the "stay small" side. This is the bias–variance tradeoff made adjustable — see Bias-Variance Tradeoff and Overfitting and Generalization.

• Too small $\lambda$ → overfitting (weights memorize noise).
• Too large $\lambda$ → underfitting (weights all near zero, network too dull).
• Tune it on a validation set.

7. Putting it together — $R(w)$, $\tilde{J}$, and $\text{sign}$

• $R(w)$ = the penalty function — a number that grows when weights grow.
• $\tilde{J}$ (tilde-J) = the new landscape the optimizer actually walks on.
• $\text{sign}(w)$ = a helper returning $+1$ if $w>0$, $-1$ if $w<0$, $0$ at zero. It appears because the slope of $|w|$ is $+1$ on the right and $-1$ on the left — a constant push whose only information is the direction toward zero. That constant-size push is the seed of L1 sparsity.

Prerequisite map

Every arrow says "you must understand the source before the target makes sense." The final node is the parent topic itself.

Related building blocks

Once these symbols are solid, the parent note connects out to: Gradient Descent (the walk downhill), Maximum a Posteriori (MAP) Estimation (the Bayesian "why" of penalties), Lasso and Ridge Regression (L1/L2 in linear models), Adam and AdamW Optimizers (why decay ≠ L2 for adaptive methods), and Dropout (a different anti-overfitting tool).

Equipment checklist

Test yourself — cover the right side and answer before revealing.