Visual walkthrough — Adam and AdamW optimizers

Everything here is the visual companion to the parent note. The maths there is correct — here we watch it happen.

---

Step 1 — The problem: one learning rate cannot fit every slope

WHAT. Imagine a landscape where "height" is the loss (how wrong the model is) and "position" is a parameter $\theta$ (a single knob in the network). Plain gradient descent reads the slope $g_t$ (the steepness under your feet, pointing uphill) and steps downhill: $\theta \leftarrow \underset{\text{where you stand}}{\underbrace{\theta }}-\underset{\begin{array}{c}\text{step size,}\\ \text{one fixed number}\end{array}}{\underbrace{\eta }}\underset{\begin{array}{c}\text{slope now}\\ ({\nabla }_{\theta }L)\end{array}}{\underbrace{g_t}}$

Here $g_t$ is the gradient — the slope, positive when the ground rises as $\theta$ grows. We subtract it so we move away from uphill, i.e. downhill. $\eta$ (Greek "eta") is the learning rate, the length of one stride.

WHY it fails. Different knobs sit on wildly different slopes. One global stride $\eta$ that is safe on a steep cliff is painfully slow on a gentle plain.

PICTURE. Two valleys side by side. The same stride $\eta$ overshoots the steep one (bounces across) and crawls in the flat one.

The fix Adam proposes: give each direction its own stride, decided by two running summaries of the gradient. We build those summaries next.

---

Step 2 — The tool we need: an Exponential Moving Average

WHAT. Each minibatch hands us a fresh, noisy slope $g_t$. We want a smooth summary that trusts recent data but does not forget instantly. That tool is the exponential moving average (EMA): $m_t=\underset{\begin{array}{c}\text{memory}\\ 0\le {\beta }_1<1\end{array}}{\underbrace{{\beta }_1}}\underset{\text{old summary}}{\underbrace{m_{t-1}}}+\underset{\text{room for new}}{\underbrace{(1-{\beta }_1)}}\underset{\text{new slope}}{\underbrace{g_t}},m_0=0.$

WHY this tool and not a plain average? A plain average over all past gradients weights the ancient first step as much as the latest — but early gradients are stale, the landscape has moved on. The EMA is a leaky bucket: each step it keeps fraction ${\beta }_1$ of the old contents and pours in fraction $(1-{\beta }_1)$ of the new. Old values fade geometrically (${\beta }_1,{\beta }_1^2,{\beta }_1^3,\dots$), so recent slopes dominate automatically. See Exponential Moving Average.

PICTURE. The leaky bucket: old water partly drains, a splash of the new gradient is added, the level (=$m_t$) is a smoothed trace of the noisy input.

We will use this same EMA machine twice — once on the gradient, once on its square.

---

Step 3 — First moment: smooth the direction (momentum)

WHAT. Feed the raw gradient into the EMA. Call the result $m_t$, the first moment: $m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,{\beta }_1=\mathrm{0.9.}$

With ${\beta }_1=0.9$ the bucket keeps $90\%$ of its old content, so $m_t$ is a heavily smoothed version of the slope — essentially an estimate of the average gradient $\mathbb{E}[g_t]$.

WHY. Noisy minibatch gradients zig-zag. Averaging cancels the random wobble and keeps only the consistent pull — the true downhill direction. This is exactly the idea behind Momentum and Nesterov Acceleration: build velocity along persistent directions.

PICTURE. A jittery cloud of raw arrows $g_1,g_2,\dots$; the smooth arrow $m_t$ threads through their average, wobble removed.

---

Step 4 — Second moment: smooth the magnitude

WHAT. Run the same EMA on the squared gradient $g_t^2$ (squaring is elementwise — each coordinate separately). Call it $v_t$, the second moment: $v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)\underset{\begin{array}{c}\text{size of slope,}\\ \text{always}\ge 0\end{array}}{\underbrace{g_t^2}},{\beta }_2=\mathrm{0.999.}$

Squaring throws away the sign and keeps only how big the slope is. So $v_t$ estimates $\mathbb{E}[g_t^2]$ — the typical magnitude/variance of the gradient in that direction. ${\beta }_2=0.999$ makes this an even slower, longer-memory average than $m_t$.

WHY. A direction with large $v_t$ is steep or noisy — we want small steps there. A direction with tiny $v_t$ is flat and reliable — we can stride there. $v_t$ is the per-direction "danger meter." This is the RMSProp idea.

PICTURE. Two coordinates: one with big bouncing squared-gradients (high $v$, red danger meter full), one with small steady ones (low $v$, meter low).

---

Step 5 — The zero-start bias, and why we must correct it

WHAT. Both buckets start empty: $m_0=v_0=0$. So on step 1 the bucket is mostly still empty water — the summary reads too small. Unrolling the recursion for a roughly constant gradient with mean $\bar{g}$: $\mathbb{E}[m_t]=\bar{g}(1-{\beta }_1^t).$

The factor $(1-{\beta }_1^t)$ is the fraction of the bucket that has actually filled by step $t$. At $t=1$ it is only $1-{\beta }_1=0.1$ — we read 10% of the truth. We repair this by dividing it back out: ${\hat{m}}_t=\frac{m_t}{1-{\beta }_1^t},{\hat{v}}_t=\frac{v_t}{1-{\beta }_2^t}.$ The hat ($\hat{}$) means "bias-corrected." Now $\mathbb{E}[{\hat{m}}_t]\approx \bar{g}$: honest.

WHY. Without this, early ${\hat{v}}_t$ is tiny, so $1/\sqrt{{\hat{v}}_t}$ explodes and the first steps are dangerously huge — exactly when training is most fragile. See Bias-Variance in Gradient Estimation.

PICTURE. The correction factor $(1-{\beta }^t)$ rising from small toward $1$; the raw curve $m_t$ crawls up, the corrected ${\hat{m}}_t$ snaps to the true value $\bar{g}$ from step one.

As $t\to \infty$, ${\beta }^t\to 0$, the factor $\to 1$, and the correction quietly switches itself off — it only matters early.

---

Step 6 — The update: a signal-to-noise ratio

WHAT. Combine everything. Move against the smoothed direction ${\hat{m}}_t$, but divide by the smoothed magnitude $\sqrt{{\hat{v}}_t}$: $\boxed{{\theta }_t={\theta }_{t-1}-\eta \frac{\stackrel{\text{avg direction}}{\overbrace{{\hat{m}}_t}}}{\underset{\text{typical size}}{\underbrace{\sqrt{{\hat{v}}_t}}}+\underset{\begin{array}{c}\text{tiny guard}\\ \approx 10^{-8}\end{array}}{\underbrace{ϵ}}}}$

We take $\sqrt{{\hat{v}}_t}$ (not ${\hat{v}}_t$) because ${\hat{v}}_t$ has units of gradient-squared; its square root has the same units as ${\hat{m}}_t$, so the ratio is a clean dimensionless number. $ϵ$ keeps the denominator from ever being $0$.

WHY. Read ${\hat{m}}_t/\sqrt{{\hat{v}}_t}$ as signal ÷ noise:

• Consistent direction: ${\hat{m}}_t\approx \sqrt{{\hat{v}}_t}$, ratio $\approx \pm 1$ → a full, confident stride of size $\eta$.
• Noisy direction: ${\hat{m}}_t\ll \sqrt{{\hat{v}}_t}$, ratio $\approx 0$ → a timid step. Adam slows down where it is unsure.

And it is scale-invariant: multiply every gradient by $c$ and both $\hat{m}$ and $\sqrt{\hat{v}}$ scale by $c$ — the ratio is unchanged. This is what SGD could never do.

PICTURE. A dial: numerator = how aligned the recent gradients are, denominator = how large/noisy they are; the needle lands near $\pm 1$ for a clean signal, near $0$ for pure noise.

---

Step 7 — Edge cases: watch every corner

WHAT. Three degenerate situations, and what the formula does in each.

1. First step ($t=1$). ${\hat{m}}_1=g_1$, ${\hat{v}}_1=g_1^2$, so the ratio $=g_1/\sqrt{g_1^2}=\mathrm{sign}(g_1)$. The step magnitude is exactly $\eta$, no matter how big or small $g_1$ is — a normalized "compass" step.
2. Steady constant gradient ($t\to \infty$). ${\hat{m}}_t\to g$, ${\hat{v}}_t\to g^2$, ratio $\to \mathrm{sign}(g)$: again magnitude $\eta$. Adam erases gradient scale in steady state.
3. Vanishing gradient ($g_t\to 0$). Then ${\hat{v}}_t\to 0$ and the guard takes over: step $\to {\hat{m}}_t/ϵ$, capped — the update does not blow up, it fades.

WHY show all three. A reader must never meet a case you skipped. Whether the gradient is huge, tiny, positive, negative, or zero, the step size stays near $\eta$ (or safely shrinks), because the denominator tracks the numerator.

PICTURE. Three mini-panels — first step, steady state, near-zero gradient — each showing the resulting step length hugging $\eta$ (except the guarded fade at zero).

---

Step 8 — AdamW: pull weight decay out of the divide

WHAT. Weight decay shrinks weights toward zero to fight overfitting. The old trick added $\lambda \theta$ into the gradient — but then it gets divided by $\sqrt{{\hat{v}}_t}$ like everything else, so heavily-gradiented weights decay less. Backwards. AdamW keeps decay outside the adaptive fraction: ${\theta }_t={\theta }_{t-1}-\eta \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}-\underset{\begin{array}{c}\text{uniform shrink,}\\ \text{same for all weights}\end{array}}{\underbrace{\eta \lambda {\theta }_{t-1}}}.$

WHY. Now every weight multiplies by the same factor $(1-\eta \lambda )$ each step — clean, gradient-independent regularization.

PICTURE. Left: Adam+L2, decay arrows scaled unevenly by $1/\sqrt{\hat{v}}$ (small-gradient weight decays far more). Right: AdamW, all decay arrows the same length.

---

The one-picture summary

Two EMAs — one of the gradient ($m_t$, direction) and one of its square ($v_t$, magnitude) — are bias-corrected, then the update takes the ratio ${\hat{m}}_t/\sqrt{{\hat{v}}_t}$, a scale-free signal-to-noise stride, and AdamW subtracts a separate uniform decay. That is the whole optimizer in one flow.

Recall Feynman retelling — the whole walkthrough in plain words

You are hiking downhill in fog. You cannot trust one footstep, so you keep two notebooks. Notebook one (==$m_t$) writes down a smoothed "which way have I been heading" — that cancels the random zig-zag. Notebook two ($v_t$==) writes down "how steep and slippery has each direction been" by averaging the squared slopes. Because both notebooks started blank, their early entries read too low, so you scale them up by a fading factor ($\hat{m},\hat{v}$ — the bias correction) that only matters in the first pages. To actually step, you take direction ÷ steepness: on a consistent gentle slope that ratio is about $\pm 1$ so you stride a full $\eta$; on a noisy steep one it is near $0$ so you inch forward. This makes your step size roughly $\eta$ everywhere, ignoring how big the raw slope is. Finally, AdamW adds one honest rule: shrink every weight by the same small amount each step, kept outside the ratio so heavy-gradient weights are not spared.

Recall Quick self-test

Steady-state Adam step on a constant gradient $g$? ::: $\eta \mathrm{sign}(g)$ — magnitude $\eta$, scale erased. Why square root over ${\hat{v}}_t$ and not ${\hat{v}}_t$ itself? ::: to match units with ${\hat{m}}_t$ so the ratio is dimensionless and scale-invariant. Where does AdamW put weight decay? ::: outside the $\sqrt{\hat{v}}$ divide, as a uniform $\eta \lambda \theta$ subtraction.

Related: Stochastic Gradient Descent · Learning Rate Schedules · 3.2.05 Adam and AdamW optimizers (Hinglish)