Adam and AdamW optimizers

WHY do we need Adam at all?

Plain SGD uses the same learning rate for every parameter: $\theta \leftarrow \theta - \eta\, g_t,\qquad g_t=\nabla_\theta L$

Problem 1 — different scales. In deep nets, some parameters get huge gradients, others tiny. One global $\eta$ is too big for some, too small for others.

Problem 2 — noisy stochastic gradients. Each minibatch gives a noisy $g_t$ . We want to average out the noise (momentum) but also normalize by how big/variable each gradient is.

Adam answers both by tracking two exponential moving averages (EMAs) per parameter.

Deriving Adam from first principles

Step 1 — First moment (momentum): smooth the direction

We EMA the gradient itself: $m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t$ Why? $m_t$ estimates $\mathbb{E}[g_t]$ — the average gradient. This cancels noise and keeps consistent directions moving.

Step 2 — Second moment: smooth the magnitude

We EMA the squared gradient (elementwise): $v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^{2}$ Why? $v_t$ estimates $\mathbb{E}[g_t^2]$ — the size/variance of the gradient in each coordinate. Large $v_t$ ⇒ that direction is steep/noisy ⇒ take smaller steps there.

Step 3 — Bias correction (the clever bit)

Because $m_0=v_0=0$ , early EMAs are biased toward zero. Let's prove the correction. Assume $g$ roughly stationary with mean $\bar g$ . Unrolling: $m_t=(1-\beta_1)\sum_{i=1}^{t}\beta_1^{\,t-i}g_i.$ Taking expectations with $\mathbb E[g_i]=\bar g$ : $\mathbb E[m_t]=\bar g\,(1-\beta_1)\sum_{i=1}^{t}\beta_1^{\,t-i}=\bar g\,(1-\beta_1)\cdot\frac{1-\beta_1^{t}}{1-\beta_1}=\bar g\,(1-\beta_1^{t}).$ So $\mathbb E[m_t]=\bar g(1-\beta_1^t)$ is too small by factor $(1-\beta_1^t)$ . Divide it out: $\hat m_t=\frac{m_t}{1-\beta_1^{t}},\qquad \hat v_t=\frac{v_t}{1-\beta_2^{t}}.$ Now $\mathbb E[\hat m_t]\approx\bar g$ — unbiased. As $t\to\infty$ , $\beta_1^t\to0$ and the correction vanishes.

Step 4 — The update

$\boxed{\;\theta_t=\theta_{t-1}-\eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}\;}$

AdamW — decoupled weight decay

Adam-with-L2 (naive): fold decay into the gradient $g_t \leftarrow \nabla L + \lambda\theta_{t-1}\;\Rightarrow\;\text{decay gets scaled by }1/\sqrt{\hat v_t}. \;❌$

AdamW: keep decay outside the adaptive machinery $\boxed{\;\theta_t=\theta_{t-1}-\eta\Big(\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}+\lambda\,\theta_{t-1}\Big)\;}$

Worked examples

Worked example Example 1 — one Adam step by hand

Params start at $\theta_0=0.5$ . Gradient $g_1=0.2$ , hyperparams $\eta=0.1$ , $\beta_1=0.9,\beta_2=0.999,\epsilon=0$ .

$m_1=0.9(0)+0.1(0.2)=0.02$ — Why? EMA of gradient, starts from 0.
$v_1=0.999(0)+0.001(0.04)=4\times10^{-5}$ — Why? EMA of $g^2=0.04$ .
$\hat m_1=0.02/(1-0.9)=0.2$ — Why? Bias correction; at $t{=}1$ it fully un-shrinks $m$ .
$\hat v_1=4\times10^{-5}/(1-0.999)=0.04$ — Why? Same, recovers $g^2$ .
step $=\eta\,\hat m_1/\sqrt{\hat v_1}=0.1\cdot0.2/0.2=0.1$ — Why? At $t{=}1$ the ratio $=\text{sign}(g)$ , so the step magnitude is exactly $\eta$ , independent of $|g|$ .
$\theta_1=0.5-0.1=0.4$ .

Insight: the very first Adam step always has magnitude $\eta$ per coordinate — a normalized "compass" step.

Common mistakes (Steel-manned)

Recall

Recall Active recall — cover the answers

What two statistics does Adam track per parameter? ⇒ EMA of the gradient ( $m_t$ ) and EMA of the squared gradient ( $v_t$ ).
Why divide by $\sqrt{\hat v_t}$ ? ⇒ to normalize each coordinate by its gradient magnitude → adaptive per-parameter learning rate.
Why bias-correct? ⇒ EMAs start at 0, biased low by factor $(1-\beta^t)$ ; dividing removes it.
What's the steady-state Adam step on constant gradient? ⇒ $\eta\,\text{sign}(g)$ .
What exactly does AdamW change? ⇒ it applies weight decay $\eta\lambda\theta$ outside the $\sqrt{\hat v}$ normalization.

Recall Feynman: explain to a 12-year-old

Imagine hiking down a hill in fog. Momentum ( $m$ ) is remembering which way you've been walking so you don't zig-zag. The magnitude tracker ( $v$ ) notices which directions are steep and slippery, so you take tiny careful steps there and big confident steps on gentle slopes. Bias correction is because at the very start you have no memory yet, so you multiply up your first guesses to make them fair. AdamW adds a rule: "every step, shrink all your things a little toward zero, equally" — so nothing grows too big and messy. Adam = smart footwork; AdamW = smart footwork plus tidy-up.

Flashcards

Adam first moment update formula

m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t

Adam second moment update formula

v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2

Adam bias-corrected estimates

\hat m_t=m_t/(1-\beta_1^t)

\hat v_t=v_t/(1-\beta_2^t)

Adam parameter update

\theta_t=\theta_{t-1}-\eta\,\hat m_t/(\sqrt{\hat v_t}+\epsilon)

Why bias correction is needed

EMAs initialized at 0 are biased toward 0 by factor

(1-\beta^t)

; dividing by it makes the estimate unbiased.

Default Adam hyperparameters

\beta_1=0.9,\ \beta_2=0.999,\ \epsilon=10^{-8}

Interpretation of

\hat m/\sqrt{\hat v}

a signal-to-noise ratio; consistent gradients → step ≈ η, noisy gradients → small step.

Steady-state Adam step on constant gradient g

\eta\,\text{sign}(g)

— magnitude independent of

|g|

What AdamW changes vs Adam+L2

applies weight decay as

\eta\lambda\theta

decoupled from (outside) the

\sqrt{\hat v}

normalization.

Why Adam+L2 decay is wrong

the

\lambda\theta

term gets divided by

\sqrt{\hat v}

, so decay becomes gradient-dependent instead of uniform.

AdamW update equation

\theta_t=\theta_{t-1}-\eta\,\hat m_t/(\sqrt{\hat v_t}+\epsilon)-\eta\lambda\theta_{t-1}

Role of

\epsilon

beyond avoiding /0

caps the max effective learning rate (

\le\eta/\epsilon

) when

\hat v

is tiny.

Connections

Stochastic Gradient Descent — the baseline Adam generalizes.
Momentum and Nesterov Acceleration — source of the first-moment term.
RMSProp — source of the second-moment normalization.
Exponential Moving Average — the core smoothing tool and its bias.
Weight Decay and L2 Regularization — what AdamW decouples.
Learning Rate Schedules — warmup pairs naturally with Adam's early instability.
Bias-Variance in Gradient Estimation — why we smooth gradients.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, plain SGD har parameter ke liye same learning rate use karta hai, jo ill-conditioned problems (kuch directions steep, kuch flat) mein bekaar hai — wo zig-zag karta hai ya diverge ho jaata hai. Adam iska smart solution hai: har parameter ke liye do cheezein track karta hai — ek momentum ( $m_t$ , gradient ka smooth average, direction batata hai) aur doosra squared gradient ka average ( $v_t$ , magnitude batata hai). Fir update karta hai $\hat m/\sqrt{\hat v}$ se, matlab har axis ko uski apni magnitude se normalize kar deta hai. Isliye steep directions mein chhota step, flat directions mein bada step — automatic adaptive learning rate.

Bias correction ka funda simple hai: $m$ aur $v$ dono zero se start hote hain, to shuru mein value bahut chhoti (biased) aati hai. Isliye $(1-\beta^t)$ se divide karke usko "un-shrink" karte hain. Yeh training ke shuruaat mein bahut zaroori hai, warna early steps blow up ho sakte hain. Steady state mein constant gradient par Adam ka step $\eta\,\text{sign}(g)$ ho jaata hai — magnitude gradient ke size se independent, sirf direction pe depend.

AdamW ek chhota par important fix hai. Normal weight decay (L2) mein tum $\lambda\theta$ ko gradient mein add karte ho, par Adam usko $\sqrt{\hat v}$ se divide kar deta hai — matlab jinke gradient bade, unpe decay kam, jinke chhote unpe zyada. Yeh galat hai! AdamW decay ko alag rakhta hai: har step $\eta\lambda\theta$ seedha subtract, sab weights pe equal shrinkage. Isiliye aajkal transformers mein AdamW default optimizer hai. Yaad rakho: Mean upar, RMS neeche, Weight-decay alag.

WHY do we need Adam at all?

Deriving Adam from first principles

Step 1 — First moment (momentum): smooth the direction

Step 2 — Second moment: smooth the magnitude

Step 3 — Bias correction (the clever bit)

Step 4 — The update

AdamW — decoupled weight decay

Worked examples

Common mistakes (Steel-manned)

Recall

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Training Deep Networks

Connections