Adam and AdamW optimizers: Question bank — AI-ML deep dive

Before you start, we build up the symbols so nothing below is a mystery.

Definition Every symbol on this page, defined

$g_{t}$ — the raw gradient at step $t$ : a noisy arrow pointing in the direction the loss increases fastest. We want to step the other way.
$η$ (eta) — the learning rate: a single scalar knob for step size. Typical default $η = 1 0^{- 3}$ .
$β_{1}$ — the decay for the direction average, in $[0, 1)$ . Default $β_{1} = 0.9$ (memory of about 10 steps).
$β_{2}$ — the decay for the magnitude average, in $[0, 1)$ . Default $β_{2} = 0.999$ (memory of about 1000 steps).
$ϵ$ (epsilon) — a tiny floor in the denominator. Default $ϵ = 1 0^{- 8}$ .
$λ$ (lambda) — the weight-decay strength (regularization). Typical $λ \approx 1 0^{- 2}$ ; $λ = 0$ means "off".
$E [\cdot]$ — the expectation (the long-run average value) of a random quantity.
$Var (\cdot)$ — the variance: how much a quantity jiggles around its mean, $Var (g) = E [g^{2}] - (E g)^{2}$ .

The four objects this whole page revolves around:

$m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}$ — the EMA of $g_{t}$ : smoothed direction (momentum, Momentum and Nesterov Acceleration).
$v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}$ — the EMA of $g_{t}^{2}$ : smoothed magnitude per coordinate (the RMSProp idea).
$\overset{m}{^}_{t} = m_{t} / (1 - β_{1}^{t}), \overset{v}{^}_{t} = v_{t} / (1 - β_{2}^{t})$ — their bias-corrected versions (both EMAs start at $0$ , so early on they read too low; dividing by $1 - β^{t}$ undoes that).
The update: $θ_{t} = θ_{t - 1} - η \overset{m}{^}_{t} / (\overset{v}{^}_{t} + ϵ)$ .

The figures below make momentum, the magnitude divide, and the signal-to-noise picture visible before you meet the traps.

Why questions

Why square the gradient for

v_{t}

instead of taking

∣ g_{t} ∣

Squaring gives an EMA of

E [g^{2}]

, a clean, differentiable, always-non-negative magnitude/variance estimate. It is only unbiased after the bias correction

\overset{v}{^}_{t} = v_{t} / (1 - β_{2}^{t})

— the raw

v_{t}

reads too low early on.

\overset{v}{^}_{t}

then recovers an RMS scale.

Why is

\overset{m}{^}_{t} / \overset{v}{^}_{t}

interpreted as a signal-to-noise ratio?

\overset{m}{^}

is the mean gradient (signal),

\overset{v}{^}

is its typical magnitude (signal + noise). Consistent directions give ratio

\approx 1

(full step); noisy directions give ratio

\approx 0

(tiny step) — see the red curve in Figure 3.

Why choose

β_{2} = 0.999

much closer to 1 than

β_{1} = 0.9

The magnitude estimate

v_{t}

should be smooth and stable over a long window (~1000 steps), so it uses a slow decay; the direction

m_{t}

should react faster to genuine changes (~10 steps), so it uses a shorter memory.

Why does AdamW decouple weight decay rather than fix the L2 gradient by pre-scaling it?

Because to pre-scale correctly you'd need

\overset{v}{^}_{t}

before computing it, a chicken-and-egg problem; decoupling sidesteps the adaptive machinery entirely and gives uniform, predictable shrinkage.

Why does Adam often converge faster than SGD early in training but sometimes generalize worse?

The magnitude normalization lets it move confidently through ill-scaled loss landscapes early, but that same normalization can overfit sharp minima — a reason plain Stochastic Gradient Descent with tuning still wins on some tasks.

Why can Adam's effective learning rate be interpreted as

η / \overset{v}{^}_{t}

and how does that interact with a schedule?

Because the update is

(η / \overset{v}{^}_{t}) \overset{m}{^}_{t}

, the schedule scales

η

on top of Adam's own per-coordinate adaptation; the two stack, so a warmup still helps even though Adam adapts internally. See Learning Rate Schedules.

Edge cases

What happens on the very first step if

g_{1} = 0

exactly?

Then

m_{1} = v_{1} = 0

, so

\overset{m}{^}_{1} = 0

and the update is

0

(with

ϵ > 0

the denominator is nonzero). No movement — Adam does nothing without gradient information.

A coordinate has

\overset{v}{^}_{t}

far smaller than

ϵ

. What limits its step, and what happens to scale-invariance?

The denominator is dominated by

ϵ

, so the step

\approx η \overset{m}{^}_{t} / ϵ

— capped near

η / ϵ

. Here scale-invariance breaks: multiply gradients by

c

and now

\overset{m}{^}

scales by

c

but the

ϵ

-dominated denominator barely moves, so the step scales by

\approx c

(SGD-like) instead of staying fixed.

Without bias correction, what goes wrong in the first ~100 steps?

\overset{v}{^}

(uncorrected) is far too small, so

1/ v

is huge and early updates explode or destabilize — exactly when training is most fragile.

A gradient is consistently zero for a while, then suddenly spikes. How does

v_{t}

respond, and what does it do to the step?

v_{t}

lags because of its slow decay (

β_{2} = 0.999

), so for a few steps

\overset{v}{^}

underestimates the spike and the step there is temporarily too large before

v_{t}

catches up.

If you set

λ

but forget you also folded L2 into the loss, what double-counts?

Decay is applied twice — once (scaled) inside

g_{t}

and once (uniformly) as the decoupled AdamW term — so weights shrink far more than intended and the model underfits.

β_{1} \to 1

with

t

fixed, what happens to the correction factor

1/ (1 - β_{1}^{t})

It diverges toward infinity, meaning

m_{t}

is almost entirely the (near-zero) initial average and needs enormous un-shrinking; this is why

β_{1}

is kept well below 1.

Recall One-line self-test

Cover everything above and answer: In steady state, what determines Adam's step magnitude? Only $η$ (and $ϵ$ ) — not the gradient size. Adam responds to direction consistency, not raw magnitude.

Question bank — Adam and AdamW optimizers

True or false — justify

Spot the error

Why questions

Edge cases