L1 - L2 weight decay in deep nets: Question bank

Reminders you'll lean on (all defined in the parent):

$w$ = a single weight; $∥ w ∥_{2}^{2} = \sum_{i} w_{i}^{2}$ (L2 penalty, summed squares); $∥ w ∥_{1} = \sum_{i} ∣ w_{i} ∣$ (L1 penalty, summed absolute values).
$λ \geq 0$ = regularization strength; $η$ = learning rate.
L2 update: $w \leftarrow (1 - η λ) w - η \nabla_{w} J$ . L1 penalty gradient: $λ sign (w)$ .
$sign (w)$ = $+ 1$ if $w > 0$ , $- 1$ if $w < 0$ : it reads off which side of zero a weight sits on, so the L1 push is always full-strength toward zero.

True or false — justify

L2 weight decay drives many weights to exactly zero.

False. L2's push is

η λ w

, which shrinks with $w$ , so it fades to nothing as

w \to 0

— weights become tiny but almost never exactly zero. Exact zeros are L1's signature.

For plain SGD, "adding an L2 penalty to the loss" and "multiplying weights by

(1 - η λ)

each step" are the same thing.

True. We derived it:

w \leftarrow w - η (\nabla_{w} J + λ w) = (1 - η λ) w - η \nabla_{w} J

. The penalty gradient

λ w

is the shrink factor.

For Adam, L2 penalty and true weight decay are still identical.

False. Adam divides each gradient (including the

λ w

term) by a per-parameter adaptive scale, so already-large-gradient weights get less decay. AdamW instead applies

(1 - η λ)

decay separately from the gradient — that decoupling is the whole point of AdamW.

Increasing

λ

always improves generalization.

False. It reduces variance up to a point, but too-large

λ

crushes all weights toward 0, giving an over-smooth function that can't fit the signal — that's underfitting (high bias). It's a tradeoff with a sweet spot found on validation data.

Bias terms should be regularized just like weights, for consistency.

False. Biases only shift the function up/down; they don't create the spiky high-variance behaviour that overfits. Penalizing them adds bias without cutting variance and can hurt the fit — so standard practice leaves them (and BatchNorm params) unpenalized.

If the data gradient is zero, an L2-regularized weight stays put.

False. With

\nabla_{w} J = 0

the update is

w \leftarrow (1 - η λ) w

, geometric decay toward 0. A weight only survives if the data gradient keeps re-supplying it.

L1 penalty corresponds to a Gaussian prior on the weights.

False. L1 ⇔ Laplace prior (sharply peaked at 0, heavy tails), which is what encourages exact zeros. L2 ⇔ Gaussian prior. This is the MAP view of regularization.

λ = 0

the regularized objective equals the plain data loss.

True.

\tilde{J} = J + λ R

with

λ = 0

leaves

\tilde{J} = J

— no penalty, so no decay and no sparsity. It's the degenerate "no regularization" case.

Spot the error

"L2 gradient is

λ sign (w)

, so it pushes with constant force."

The L1 gradient is

λ sign (w)

(constant force). L2's gradient is

λ w

— proportional to the weight, so the force weakens as

w

shrinks. Swapping these two is the core L1-vs-L2 confusion.

"Because

(1 - η λ) < 1

, weights decay, so I can pick any large

η λ

to decay faster and it's always fine."

η λ > 1

then

(1 - η λ)

is negative and

∣1 - η λ ∣

can exceed 1, so weights flip sign and blow up each step. The decay-toward-zero picture only holds for

0 < η λ < 1

"L1 gives small weights and L2 gives sparse weights."

Backwards. L1 = sparse (exact zeros) thanks to the constant-force gradient; L2 = uniformly small but rarely zero. Compare with Lasso (L1) and Ridge (L2).

"The equilibrium weight

w^{⋆} = - \nabla_{w} J / λ

means bigger

λ

makes weights bigger."

Bigger

λ

appears in the denominator, so it makes

∣ w^{⋆} ∣

smaller — stronger regularization means the data must supply a larger gradient to justify the same weight.

"L2 is called weight decay because it decays the learning rate over time."

No — the learning rate

η

is untouched. It's the weights that decay: each step multiplies

w

(1 - η λ) < 1

. Learning-rate schedules are a separate mechanism.

"Weight decay and Dropout do the same thing, so use only one."

Both fight overfitting but by different mechanisms — decay shrinks weight magnitudes, dropout randomly removes units to prevent co-adaptation. They're complementary and often combined; neither replaces the other.

Why questions

Why does L1 produce exact zeros while L2 does not?

L1's push

η λ sign (w)

stays full-strength no matter how small

w

is, so it can carry a small weight across zero and a soft-threshold pins it at 0. L2's push

η λ w

vanishes as

w \to 0

, so it slows to a crawl and never lands exactly on zero.

Why is it fair to say "the data must earn every unit of weight"?

At equilibrium the decay force

λ w

exactly cancels the data gradient

\nabla_{w} J

, giving

w^{⋆} = - \nabla_{w} J / λ

. A weight is only nonzero to the extent the data gradient keeps pushing back — no useful gradient, no surviving weight.

Why do smaller weights tend to generalize better?

Small weights make the network's function flatter and smoother, so it can't carve sharp spikes to memorize noise. Smoother functions have lower variance and thus better generalization.

Why is adding a penalty equivalent to a Bayesian prior?

MAP maximizes

lo g P (data ∣ w) + lo g P (w)

; the data term is the negative loss and the log-prior term becomes the penalty. So

λ R (w)

is literally a statement of belief that most weights should be near zero.

Why does AdamW exist if Adam can already add an L2 term?

Under Adam, the

λ w

term is scaled by each parameter's adaptive learning rate, so decay strength varies unpredictably per weight. AdamW decouples decay, applying a clean

(1 - η λ)

shrink independent of the gradient scaling, restoring the intended uniform decay.

Why do we penalize

\frac{1}{2} ∥ w ∥_{2}^{2}

(with the

\frac{1}{2}

) rather than

∥ w ∥_{2}^{2}

The

\frac{1}{2}

is a convenience: differentiating

\frac{1}{2} w^{2}

gives exactly

w

(the 2 cancels), so the gradient is the clean

λ w

with no stray factor. It doesn't change what's optimized, only tidies the algebra.

Edge cases

What happens to a weight sitting exactly at

w = 0

under L1?

sign (0)

is undefined, so the derivative doesn't exist there; the subgradient lets any value in

[- 1, 1]

apply. In practice the soft-threshold operator simply holds it at 0 unless the data gradient is strong enough to pull it out.

What is the update when

λ = 0

It reduces to ordinary Gradient Descent:

w \leftarrow w - η \nabla_{w} J

, with no shrink factor and no sparsity pressure. Regularization is fully switched off.

What if the data gradient exactly equals the decay force?

The weight is at its equilibrium

w^{⋆} = - \nabla_{w} J / λ

and stays there — the two opposing pushes cancel, so the weight neither grows nor decays.

What if

η λ = 1

exactly (with zero data gradient)?

The shrink factor

(1 - η λ) = 0

, so

w

is set to 0 in a single step. Above 1 it goes negative and oscillates/diverges — so

η λ = 1

is the boundary of stable decay.

Does weight decay affect a weight whose data gradient is enormous and constant?

Yes, but only mildly: it settles at

w^{⋆} = - \nabla_{w} J / λ

, which is large when

∣ \nabla_{w} J ∣

is large. Strong, consistent data evidence overpowers the decay, so important weights stay big — exactly the desired behaviour.

What does L2 do to two weights that are duplicates (same feature)?

L2 spreads the load, shrinking both to share the total evenly (each ends up small), whereas L1 tends to keep one and zero the other. This is why L2 is "spread shrinkage" and L1 is "selection."

Recall One-line self-test

If you can state, for any item above, both the verdict AND the mechanism ( $λ w$ vs $λ sign (w)$ , and $(1 - η λ)$ ), you own this topic.