Adam and AdamW optimizers: Worked examples — AI-ML deep dive

This page is the worked-examples companion to Adam and AdamW optimizers. The parent note derived the formulas; here we use them on every kind of input the algorithm can meet — the first step, later steps, zero gradients, huge gradients, sign flips, decay, and a couple of exam traps. By the end you should never hit a case you haven't seen.

Everything below leans on four moving parts you already met in the parent note. Let's re-anchor them in one place so no symbol is unexplained.

Think of $\overset{m}{^}_{t} / \overset{v}{^}_{t}$ as a compass: it mostly reports which way to move, and its length hovers near 1 rather than tracking how big the gradient is. Keep that picture — most surprises below come from it.

The scenario matrix

Every situation Adam can face falls into one of these cells. The examples that follow are tagged with the cell they cover.

Cell	What's special about the input	Covered by
A. First step	$t = 1$ , EMAs still empty	Example 1
B. Positive vs negative gradient	sign of $g$	Example 1 (+), Example 2 (−)
C. Steady state, constant $g$	$t \to \infty$ , gradient stable	Example 3
D. Huge vs tiny gradient, same direction	magnitude extremes	Example 4
E. Zero / degenerate gradient	$g = 0$ , or $g$ suddenly vanishes	Example 5
F. Sign flip / oscillating gradient	$g$ changes sign each step	Example 6
G. AdamW vs Adam+L2 decay	$λ > 0$ , decoupled vs coupled	Example 7
H. Early-step instability without bias correction	$t$ small, correction on/off	Example 8
I. Word problem (real training)	choosing $η, λ$ in practice	Example 9
J. Exam twist	$ϵ$ dominating, effective-LR cap	Example 10

Prerequisites worth a glance if any line feels unfamiliar: Exponential Moving Average, Momentum and Nesterov Acceleration, RMSProp, Stochastic Gradient Descent, Weight Decay and L2 Regularization, Bias-Variance in Gradient Estimation.

Worked examples

Worked example Example 1 — The very first step (Cells A, B+)

Start at $θ_{0} = 0.5$ , gradient $g_{1} = 0.2$ , with $η = 0.1$ , defaults, and $ϵ = 0$ for clarity.

Forecast: guess the step size before reading on. Does a gradient of only $0.2$ give a tiny step?

$m_{1} = 0.9 (0) + 0.1 (0.2) = 0.02$ . Why this step? $m$ is an EMA that begins at $0$ ; one gradient nudges it a little.
$v_{1} = 0.999 (0) + 0.001 (0. 2^{2}) = 4 \times 1 0^{- 5}$ . Why? Same EMA, but on the square $0.04$ .
$\overset{m}{^}_{1} = 0.02/ (1 - 0.9) = 0.2$ . Why? At $t = 1$ the factor $1 - β_{1}^{1} = 0.1$ exactly cancels the $(1 - β_{1})$ shrink — bias correction fully restores the raw gradient.
$\overset{v}{^}_{1} = 4 \times 1 0^{- 5} / (1 - 0.999) = 0.04$ . Why? Restores the raw $g^{2}$ .
step $= η \overset{m}{^}_{1} / \overset{v}{^}_{1} = 0.1 \cdot 0.2/0.2 = 0.1$ . Why? $\overset{m}{^} / \overset{v}{^} = g /∣ g ∣ = sign (g) = + 1$ ; the compass length is exactly $1$ .
$θ_{1} = 0.5 - 0.1 = 0.4$ .

Verify: the step magnitude is $η = 0.1$ , not $η ∣ g ∣ = 0.02$ . The first Adam step is a normalized unit-compass step whose size ignores $∣ g ∣$ . ✓

Worked example Example 3 — Steady state on a constant gradient (Cell C)

Feed the same gradient $g = 0.3$ forever. Find the limiting per-step size (take $ϵ \to 0$ ).

Forecast: as the EMAs settle, does the step keep shrinking, grow, or lock onto a fixed value?

As $t \to \infty$ an EMA of a constant equals that constant: $m_{t} \to g$ , $v_{t} \to g^{2}$ . Why? $m_{\infty} = β_{1} m_{\infty} + (1 - β_{1}) g \Rightarrow m_{\infty} = g$ .
$β_{1}^{t}, β_{2}^{t} \to 0$ , so bias correction vanishes: $\overset{m}{^}_{t} \to g$ , $\overset{v}{^}_{t} \to g^{2}$ . Why? The $(1 - β^{t})$ denominators both go to $1$ .
step $\to η g / g^{2} = η sign (g)$ . Why? $g^{2} = ∣ g ∣$ , so $g /∣ g ∣ = sign (g)$ .

Verify: with $g = 0.3, η = 0.1$ the steady step is $0.1$ , and it would be the same $0.1$ for $g = 300$ . Adam has normalized gradient scale completely away in steady state. ✓

What the figure shows: the plot below traces the Adam step magnitude against the step count $t$ for three constant gradients — $g = 0.3$ (teal), $g = 30$ (orange), $g = 0.001$ (plum). Read three things off it. (1) All three curves start high or low but bend toward the same dashed line at $η = 0.1$ — that is the $η sign (g)$ plateau this example predicts. (2) The convergence takes hundreds of steps, driven by the slow $β_{2} = 0.999$ memory. (3) The final height is identical regardless of whether the gradient was $0.001$ or $30$ — visual proof that Adam erases gradient scale in steady state.

Worked example Example 5 — Zero / vanishing gradient (Cell E)

A parameter has been getting $g = 0.4$ for many steps (so $v$ has grown), then suddenly $g = 0$ for one step. State $t = 1000$ , $\overset{v}{^}_{999} \approx 0.16$ , $η = 0.01$ , $ϵ = 1 0^{- 8}$ .

Forecast: does the step instantly become $0$ ?

New $m_{1000} = 0.9 m_{999} + 0.1 (0)$ : momentum still carries the old direction, decayed once. Why? EMA has memory; one zero doesn't erase it. If $m_{999} \approx 0.4$ then $m_{1000} \approx 0.36$ .
New $v_{1000} = 0.999 v_{999} + 0.001 (0) \approx v_{999} \approx 0.16$ . Why? $β_{2} = 0.999$ means the magnitude memory barely moves.
step $\approx η \overset{m}{^} / \overset{v}{^} \approx 0.01 \cdot 0.36/0.4 \approx 0.009$ . Why? Still a near-full compass step, because momentum remembers where we were heading.
Contrast: if $g$ had been $0$ from the very start ( $m = v = 0$ ), then step $= η \cdot 0/ (0 + ϵ) = 0$ . Why? Nothing to move on; the $ϵ$ floor keeps it from dividing $0/0$ .

Verify: a momentary zero gradient still gives $\approx 0.009$ , not $0$ — Adam glides through flat spots. A permanent zero gives exactly $0$ . Both cases handled without blow-up thanks to $ϵ$ . ✓

Worked example Example 6 — Oscillating (sign-flipping) gradient (Cell F)

A parameter sits in a sharp valley: gradient alternates $g = + 1, - 1, + 1, - 1, \dots$ . What does Adam do? $η = 0.01$ .

Forecast: with the direction flipping every step, does Adam keep taking full $\pm η$ steps or damp itself?

$v$ tracks $g^{2} = 1$ regardless of sign, so $\overset{v}{^} \to 1$ , $\overset{v}{^} \to 1$ . Why? Squaring erases the flips — the magnitude tracker sees a steady $1$ .
$m$ averages $+ 1$ and $- 1$ , so $\overset{m}{^} \to 0$ . Why? Opposite-sign values cancel in the EMA; the direction signal is washed out.
step $= η \overset{m}{^} / \overset{v}{^} \to η \cdot 0/1 = 0$ . Why? Zero net direction ⇒ (near) zero step.

Verify: in a bouncing direction the step shrinks toward $0$ — Adam refuses to oscillate wildly across a valley, unlike a large fixed- $η$ SGD step that would ping-pong. This is the "signal-to-noise" intuition made concrete: high magnitude, low mean ⇒ small step. ✓

Worked example Example 7 — AdamW vs Adam+L2 on two weights (Cell G)

Two weights, both at $θ = 1$ , both with decay $λ = 0.01$ , $η = 1$ . Weight $A$ has tiny gradients so $\overset{v}{^}_{A} \approx 0.01$ ; weight $B$ has big gradients so $\overset{v}{^}_{B} \approx 1$ . Compare the decay contribution only.

Forecast: with the same $λ$ and $θ$ , do both weights shrink by the same amount under each method?

Adam + L2 folds $λ θ$ into the gradient, so its decay term is later divided by $\overset{v}{^}$ : decay $= η λ θ / \overset{v}{^}$ .
- $A$ : $1 \cdot 0.01 \cdot 1/0.01 = 1.0$ .
- $B$ : $1 \cdot 0.01 \cdot 1/1 = 0.01$ . Why bad? $A$ decays 100× more than $B$ — decay strength depends on unrelated gradient history.
AdamW keeps decay outside the $\overset{v}{^}$ machinery: decay $= η λ θ$ .
- $A$ : $0.01$ . $B$ : $0.01$ . Why good? Uniform shrink factor $(1 - η λ)$ for every weight — genuine weight decay.

Verify: AdamW gives $0.01$ for both; Adam+L2 gives $1.0$ vs $0.01$ , a 100× discrepancy. The two are provably not the same whenever $λ > 0$ and gradients differ. ✓

Worked example Example 8 — Early-step instability without bias correction (Cell H)

At $t = 1$ , $g_{1} = 0.2$ , defaults, $ϵ = 0$ . Compute the step with and without bias correction.

Forecast: the correction "vanishes as $t \to \infty$ ", so at $t = 1$ is it a tiny cosmetic tweak?

Uncorrected: $m_{1} = 0.02$ , $v_{1} = 4 \times 1 0^{- 5}$ , step $= η m_{1} / v_{1} = 0.1 \cdot 0.02/ 4 \times 1 0^{- 5}$ . Why? Use the raw EMAs. $4 \times 1 0^{- 5} \approx 0.006325$ .
That gives step $\approx 0.1 \cdot 0.02/0.006325 \approx 0.316$ . Why alarming? A step of $0.316$ from a gradient of $0.2$ — over 3× the intended $η$ .
Corrected (Example 1): step $= 0.1$ exactly. Why? Correction divides $m$ by $0.1$ but $v$ by $0.001 \approx 0.0316$ ; the mismatch is what fixes the ratio.

Verify: uncorrected $\approx 0.316$ vs corrected $= 0.1$ — the correction is a 3.16× factor at $t = 1$ , exactly $1/ 1 - β_{2} = 1/ 0.001$ . Far from cosmetic when training is most fragile. ✓

Worked example Example 9 — Word problem: picking

$η$ and $λ$ (Cell I) You fine-tune a transformer with AdamW. You want (a) each weight to decay by 1% per step independent of gradients, and (b) the raw update step near steady state to be about $3 \times 1 0^{- 4}$ . Which $η, λ$ ?

Forecast: are $η$ and $λ$ independent knobs here, or do they interact?

Steady raw update magnitude $\approx η \cdot ∣ sign (g) ∣ = η$ (Example 3). So set $η = 3 \times 1 0^{- 4}$ . Why? In AdamW the adaptive part contributes $\approx η$ per step; that's your learning-rate budget.
Decay factor per step is $(1 - η λ)$ . Want it $= 0.99$ , i.e. $η λ = 0.01$ . Why? AdamW subtracts $η λ θ$ , a fractional shrink of $η λ$ .
Solve $λ = 0.01/ η = 0.01/ (3 \times 1 0^{- 4}) \approx 33.3$ . Why? Decouple: because decay is $η λ θ$ , a small $η$ forces a large $λ$ for the same shrink.

Verify: $η λ = 3 \times 1 0^{- 4} \cdot 33.33 = 0.01$ ✓ ⇒ shrink factor $0.99$ ✓, and steady step $η = 3 \times 1 0^{- 4}$ ✓. (In practice frameworks report the product $η λ$ as the effective decay, which is why AdamW decay must be retuned whenever you change $η$ — see Learning Rate Schedules.)

Worked example Example 10 — Exam twist:

$ϵ$ dominates (Cell J) A brand-new parameter fires once with a microscopic gradient $g_{1} = 1 0^{- 6}$ . Defaults, $ϵ = 1 0^{- 8}$ , $η = 1 0^{- 3}$ . What is the actual step? Does the compass still have length $1$ ?

Forecast: Example 1 said the first step is always $η$ . Is that exactly true here?

$\overset{m}{^}_{1} = g_{1} = 1 0^{- 6}$ , $\overset{v}{^}_{1} = g_{1}^{2} = 1 0^{- 12}$ , so $\overset{v}{^}_{1} = 1 0^{- 6}$ . Why? Bias correction restores raw values at $t = 1$ .
Now compare $\overset{v}{^}_{1} = 1 0^{- 6}$ with $ϵ = 1 0^{- 8}$ : here $\overset{v}{^}_{1} ≫ ϵ$ , so $ϵ$ is negligible and step $\approx η = 1 0^{- 3}$ . Why check? $ϵ$ only bites when $\overset{v}{^}_{1} ≲ ϵ$ .
Push to the extreme: if instead $g_{1} = 1 0^{- 10}$ , then $\overset{v}{^}_{1} = 1 0^{- 10} ≪ ϵ = 1 0^{- 8}$ , and step $= η \overset{m}{^} / (\overset{v}{^} + ϵ) \approx η \cdot 1 0^{- 10} /1 0^{- 8} = η \cdot 1 0^{- 2} = 1 0^{- 5}$ . Why? When $\overset{v}{^}$ is negligible next to $ϵ$ , the denominator freezes at $ϵ$ and the step becomes $η \overset{m}{^} / ϵ$ — which grows with $\overset{m}{^}$ but stays bounded because $\overset{m}{^}$ itself is tiny here.
What is the largest step $ϵ$ permits? Set $\overset{v}{^} \to 0$ so the denominator is just $ϵ$ : then step $= η \overset{m}{^} / ϵ$ . Since a single coordinate's $∣ \overset{m}{^} ∣ \leq \overset{v}{^}$ can never exceed... more simply, in the extreme $\overset{v}{^} \to 0$ with $\overset{m}{^} \to 0$ together (as here, $\overset{m}{^} = g$ , $\overset{v}{^} = ∣ g ∣$ ), the ratio $\overset{m}{^} / (\overset{v}{^} + ϵ) = g / (∣ g ∣ + ϵ) \leq 1$ , so the compass length is at most 1 and the step is at most $η$ . Why this matters: $ϵ$ prevents the tiny-gradient case from blowing up the step; it caps the compass at length 1, not above it.

Verify: with $g_{1} = 1 0^{- 6}$ step $\approx 1 0^{- 3} = η$ (compass $\approx 1$ ); with $g_{1} = 1 0^{- 10}$ step $\approx 1 0^{- 5}$ (compass $= 1 0^{- 10} / (1 0^{- 10} + 1 0^{- 8}) \approx 1 0^{- 2}$ ). So "first step $= η$ " holds only while $\overset{v}{^} ≫ ϵ$ ; once $ϵ$ dominates it shrinks the step below $η$ . The exam trap is forgetting $ϵ$ is a real hyperparameter, not just a divide-by-zero guard. ✓

Recall

Recall Cover the answers — one line each

First-step size regardless of |g| ::: $η$ (compass length 1), as long as $\overset{v}{^} ≫ ϵ$ . Steady-state step on constant g ::: $η sign (g)$ , independent of $∣ g ∣$ . What an oscillating plus/minus gradient does to the step ::: drives $\overset{m}{^} \to 0$ so the step $\to 0$ (Adam refuses to ping-pong). Momentary zero gradient at step 1000 (m about 0.36, sqrt vhat about 0.4, eta 0.01) ::: step $\approx 0.009$ — momentum carries it through. AdamW decay for a weight at theta=1, eta=1, lambda=0.01 ::: $0.01$ (uniform); Adam+L2 gives $0.01/ \overset{v}{^}$ (gradient-dependent), e.g. $1.0$ vs $0.01$ . Bias-correction size at t=1 on the v path ::: a $\approx 3.16 \times$ step change ( $1/ 1 - β_{2} = 1/ 0.001$ ). When does epsilon actually change the step ::: when $\overset{v}{^} ≲ ϵ$ ; then it shrinks the compass below length 1, keeping the step $\leq η$ .