Momentum and Nesterov momentum: Worked examples

Before anything, let us re-anchor the two machines in plain words. Each symbol is earned here so you never have to scroll up.

Throughout, our test loss is the simplest bowl, $L (θ) = \frac{1}{2} θ^{2}$ , whose gradient is $\nabla L (θ) = θ$ . A bowl is chosen because its gradient literally equals the position, so you can read off the slope by looking at where the bead sits — no calculus needed each line. (This bowl is the 1-D face of the ill-conditioning story.)

The scenario matrix

Every situation this topic throws at you falls into one of these cells. Each worked example below is tagged with the cell(s) it covers.

#	Cell class	What is special	Covered by
A	Consistent gradients, same sign	velocity accumulates, effective step grows	Ex 1
B	Alternating gradients (steep valley)	velocity cancels, stays small	Ex 2
C	Nesterov look-ahead on a bowl	look-ahead gradient is smaller, gentler step	Ex 3
D	Zero / degenerate input ( $g = 0$ or $v = 0$ )	update collapses to a known simpler case	Ex 4
E	Limiting value $β \to$ high, steady $g$	effective LR $η / (1 - β)$ , saturation	Ex 5
F	Overshoot: $β$ too heavy	ball orbits the minimum, sign of $θ$ flips	Ex 6
G	Negative-side start / sign symmetry	mirror image, signs flip cleanly	Ex 7
H	Real-world word problem	translate a story into $η, β, g$	Ex 8
I	Exam twist: classical vs Nesterov, same seed	show they diverge numerically	Ex 9

The worked examples

Worked example Ex 1 — Cell A: consistent gradients accumulate

Bowl $L = \frac{1}{2} θ^{2}$ , $η = 0.1$ , $β = 0.9$ , start $θ_{0} = 1$ , $v_{0} = 0$ . Do 3 classical steps.

Forecast: guess — does momentum move faster or slower than plain gradient descent (which would give $θ_{1} = 0.9, θ_{2} = 0.81, θ_{3} = 0.729$ )?

$g_{1} = \nabla L (θ_{0}) = 1$ . Velocity $v_{1} = 0.9 (0) + 1 = 1$ . Step: $θ_{1} = 1 - 0.1 (1) = 0.9$ . Why this step? $v_{0} = 0$ , so the first move is identical to plain GD — there is no history yet to remember.
$g_{2} = \nabla L (θ_{1}) = 0.9$ . $v_{2} = 0.9 (1) + 0.9 = 1.8$ . $θ_{2} = 0.9 - 0.1 (1.8) = 0.72$ . Why this step? Both gradients pointed the same way (positive), so the memory term $0.9 (1)$ adds to the fresh gradient $0.9$ , giving $v_{2} = 1.8$ — the bead moves further than plain GD's $0.81$ .
$g_{3} = \nabla L (θ_{2}) = 0.72$ . $v_{3} = 0.9 (1.8) + 0.72 = 2.34$ . $θ_{3} = 0.72 - 0.1 (2.34) = 0.486$ . Why this step? Velocity keeps growing ( $1 \to 1.8 \to 2.34$ ); the bead is now well ahead of plain GD's $0.729$ .

Verify (sharper than the definition): the gap to the minimum shrinks faster for momentum. Momentum's distance-from-zero after 3 steps is $0.486$ ; plain GD's is $0.729$ . The ratio $0.486/0.729 = 0.667$ means momentum's remaining gap is only two-thirds of plain GD's remaining gap — i.e. momentum has already eaten away an extra $\frac{1}{3}$ of plain GD's leftover distance. That extra closure is exactly the accumulated velocity $v_{3} = 2.34$ doing work the raw gradient $0.72$ alone could not. ✓

Look at figure s01: the violet dashed curve is plain GD, the magenta solid curve is momentum. Notice how the two start on top of each other at step 1 (no memory yet), then the magenta curve peels below the violet one — that widening gap is the accumulated velocity. The orange dotted line is the target $θ = 0$ ; watch how much sooner magenta approaches it.

Worked example Ex 2 — Cell B: alternating gradients cancel

A single coordinate crossing a steep valley feeds gradients $g_{1} = + 1, g_{2} = - 1, g_{3} = + 1, g_{4} = - 1$ (a stand-in for the bead bouncing wall to wall). $η = 0.1$ , $β = 0.9$ , $v_{0} = 0$ . Track velocity only.

Forecast: will the velocity blow up to $\pm$ huge, or stay small?

$v_{1} = 0.9 (0) + 1 = 1$ . Why? first gradient, nothing to cancel yet.
$v_{2} = 0.9 (1) + (- 1) = - 0.1$ . Why this step? the fresh $g_{2} = - 1$ nearly wipes out the remembered $+ 0.9$ — opposite signs cancel.
$v_{3} = 0.9 (- 0.1) + 1 = 0.91$ . Why? memory is now tiny, so the new $g_{3} = + 1$ dominates but is damped.
$v_{4} = 0.9 (0.91) + (- 1) = - 0.181$ . Why? again the swing is largely cancelled.

Verify (quantify the cancellation): the raw gradients each have magnitude $1$ , so plain GD moves a fixed $η ∣ g ∣ = 0.1 \times 1 = 0.1$ every step in the flip-flopping direction — pure wasted bouncing. Momentum's actual displacements are $η ∣ v_{t} ∣ = 0.1, 0.01, 0.091, 0.0181$ . After the warm-up the momentum step ( $0.0181$ ) is about $5 \times$ smaller than plain GD's fixed $0.1$ : the oscillation energy is being averaged away, not spent. ✓

Look at figure s02: the violet bars are the raw $\pm 1$ gradients (they never shrink); the magenta bars are the velocity $v_{t}$ . The two orange dotted guides at $+ 1$ and $- 1$ mark the band the velocity never escapes — that boundedness is the cancellation. See how the magenta bar at step 2 collapses almost to zero while the violet bar is still at full height.

Worked example Ex 3 — Cell C: Nesterov look-ahead uses a smaller gradient

Same bowl, $η = 0.1$ , $β = 0.9$ , $θ_{0} = 1$ , $v_{0} = 0$ . Two Nesterov steps.

Forecast: at step 2, will Nesterov read a gradient bigger or smaller than classical (which read $0.9$ at $θ_{1}$ )?

Look-ahead $\tilde{θ}_{1} = 1 - 0.1 (0.9) (0) = 1$ . Gradient $\nabla L (\tilde{θ}_{1}) = 1$ . $v_{1} = 0.9 (0) + 1 = 1$ . $θ_{1} = 1 - 0.1 (1) = 0.9$ . Why this step? $v_{0} = 0$ so look-ahead = current point; step 1 matches classical.
Look-ahead $\tilde{θ}_{2} = 0.9 - 0.1 (0.9) (1) = 0.81$ . Gradient $\nabla L (\tilde{θ}_{2}) = 0.81$ . $v_{2} = 0.9 (1) + 0.81 = 1.71$ . $θ_{2} = 0.9 - 0.1 (1.71) = 0.729$ . Why this step? Nesterov peeked ahead to $0.81$ , where the slope is gentler ( $0.81 < 0.9$ ), so it built a smaller velocity ( $1.71$ ) than classical's $1.8$ and did not lunge as far — less overshoot risk.

Verify (isolate the correction term): the only difference between the two methods at step 2 is the gradient used — classical reads $0.9$ , Nesterov reads $0.81$ . That difference $0.9 - 0.81 = 0.09$ is exactly $η β g_{1} = 0.1 \times 0.9 \times 1 = 0.09$ , the size of the look-ahead displacement. So the "anticipatory damping" is not hand-waving: it is a concrete $0.09$ shaved off the gradient because Nesterov looked one $η β v$ step forward. ✓

Look at figure s03: the navy curve is the loss bowl. The magenta dot sits at the current point $θ_{1} = 0.9$ (slope $0.9$ ); the violet arrow "peek" carries you left to the orange dot at $0.81$ , whose slope is visibly gentler because it is lower on the bowl. The whole trick is reading the slope at the orange dot, not the magenta one.

Worked example Ex 4 — Cell D: degenerate / zero inputs

Three sub-cases. $η = 0.1, β = 0.9$ throughout.

Forecast: what should happen if you are already at the minimum with no velocity?

(D1) $θ_{t} = 0$ , $v_{t - 1} = 0$ . $g_{t} = \nabla L (0) = 0$ . $v_{t} = 0.9 (0) + 0 = 0$ . $θ_{t + 1} = 0 - 0 = 0$ . Bead stays put forever. Why? zero gradient and zero memory ⇒ nothing to move it — a genuine fixed point. ✓

(D2) $θ_{t} = 0$ but $v_{t - 1} = 2$ (arrived at the bottom still moving). $g_{t} = 0$ . $v_{t} = 0.9 (2) + 0 = 1.8$ . $θ_{t + 1} = 0 - 0.1 (1.8) = - 0.18$ . Why this step? even with zero gradient, leftover velocity carries the bead past the minimum — this is the seed of overshoot (see Ex 6).

(D3) $β = 0$ . Then $v_{t} = 0 (v_{t - 1}) + g_{t} = g_{t}$ , so $θ_{t + 1} = θ_{t} - η g_{t}$ — the rule collapses to plain Gradient Descent. Why? $β$ is the memory weight; kill it and there is no memory. This proves momentum contains plain GD as the $β = 0$ case. ✓

Verify (each sub-case is a boundary of the dynamics): D1 is the stable fixed point ( $θ = 0, v = 0$ maps to itself). D2 shows that fixed point is only reached if velocity is also zero — nonzero velocity at $θ = 0$ ejects you to $- 0.18$ , proving zero-gradient is necessary but not sufficient to stop. D3 recovers vanilla GD, so momentum is a strict generalisation. Together they bracket every degenerate input. ✓

Worked example Ex 5 — Cell E: limiting effective step for steady gradient

Feed a constant gradient $g_{t} = 1$ forever (a perfectly straight downhill), $β = 0.9$ , $v_{0} = 0$ . What does $v$ approach?

Forecast: guess the ceiling velocity before computing.

$v_{1} = 1$ , $v_{2} = 0.9 (1) + 1 = 1.9$ , $v_{3} = 0.9 (1.9) + 1 = 2.71$ , $v_{4} = 0.9 (2.71) + 1 = 3.439$ . Why this step? each step adds $1$ but keeps $90%$ of the last — a geometric build-up.
The fixed point solves $v = 0.9 v + 1 \Rightarrow 0.1 v = 1 \Rightarrow v = 10$ . So $v \to \frac{g}{1 - β} = \frac{1}{1 - 0.9} = 10$ . Why this step? at equilibrium the velocity stops changing; setting $v_{t} = v_{t - 1}$ gives the geometric-series sum.
Effective step $= η v = 0.1 \times 10 = 1$ , i.e. effective learning rate $\frac{η}{1 - β} = \frac{0.1}{0.1} = 1$ — a 10× amplification of $η$ .

Verify (convergence, not just the limit): the error to the ceiling is $10 - v_{t}$ , which reads $9, 8.1, 7.29, 6.561$ — each is exactly $0.9 \times$ the previous. So the velocity approaches its cap geometrically at rate $β = 0.9$ , taking about $ln (0.01) / ln (0.9) \approx 44$ steps to get within $1%$ . This is the same Exponentially Weighted Moving Average decay that governs how long momentum needs to "warm up." ✓

Look at figure s04: the magenta curve is the velocity climbing step by step; the orange dashed line is the ceiling $g / (1 - β) = 10$ . Notice the curve bends over and hugs the ceiling rather than crossing it — that flattening is the geometric $0.9$ -decay of the error closing the gap.

Worked example Ex 6 — Cell F:

$β$ too heavy ⇒ orbit / overshoot Bowl, $η = 0.1$ , but $β = 0.99$ (very heavy ball), $θ_{0} = 1$ , $v_{0} = 0$ . Track $θ$ until its sign flips.

Forecast: will the bead settle at $0$ , or shoot past it — and if so, at roughly which step?

$g_{1} = 1$ , $v_{1} = 0.99 (0) + 1 = 1$ , $θ_{1} = 1 - 0.1 (1) = 0.9$ .
$g_{2} = 0.9$ , $v_{2} = 0.99 (1) + 0.9 = 1.89$ , $θ_{2} = 0.9 - 0.189 = 0.711$ .
$g_{3} = 0.711$ , $v_{3} = 0.99 (1.89) + 0.711 = 2.5821$ , $θ_{3} = 0.711 - 0.25821 = 0.4528$ (exact $0.45279$ ). Why this step? velocity is climbing ( $1 \to 1.89 \to 2.58$ ) faster than the gradient shrinks — the brakes are too weak.
Continuing the same recursion (kept exact, displayed to 4 dp): $θ_{4} = 0.1620$ , $θ_{5} = - 0.1521$ . At step 5 the sign flips — the bead has crossed the minimum to the negative side. Why this step? the remembered velocity is still strongly positive even though $θ$ has almost reached $0$ , so it drives the bead straight through into negative territory.
It does not stop there: $θ_{6} = - 0.4362$ , $θ_{7} = - 0.6553$ — the bead now swings out the other side. This back-and-forth widening is the orbit. Why this step? effective LR $= η / (1 - β) = 0.1/0.01 = 10$ , wildly too big for this bowl; the heavy ball cannot decelerate in time.

Verify (why it must overshoot, not just "it does"): the safe effective LR for the bowl $L = \frac{1}{2} θ^{2}$ (gradient slope $= 1$ ) is below $2$ ; here it is $10$ , a factor $5$ over the stability edge. Compare Ex 1's $β = 0.9$ giving effective LR $1$ (stable, monotone fall). So the sign-flip at step 5 is forced by $η / (1 - β) = 10 > 2$ , not an accident of these particular numbers. ✓

Look at figure s05: the magenta curve is $θ_{t}$ for the heavy ball; watch it dive, cross the orange dotted zero-line at step 5, and swing to the far negative side — a growing orbit, never parking. The violet dashed curve is the well-behaved $β = 0.9$ run from Ex 1 for contrast, which settles calmly toward zero.

Worked example Ex 7 — Cell G: negative-side start, sign symmetry

Same bowl, $η = 0.1$ , $β = 0.9$ , but start on the left: $θ_{0} = - 1$ , $v_{0} = 0$ . Two classical steps.

Forecast: should this be an exact mirror image of Ex 1?

$g_{1} = \nabla L (- 1) = - 1$ . $v_{1} = 0.9 (0) + (- 1) = - 1$ . $θ_{1} = - 1 - 0.1 (- 1) = - 0.9$ . Why this step? on the left, the slope points down-left, i.e. negative, so the update adds $0.1$ — the bead moves right, toward $0$ .
$g_{2} = \nabla L (- 0.9) = - 0.9$ , $v_{2} = 0.9 (- 1) + (- 0.9) = - 1.8$ , $θ_{2} = - 0.9 - 0.1 (- 1.8) = - 0.72$ .

Verify (exact odd symmetry): results $- 0.9, - 0.72$ are the exact negatives of Ex 1's $0.9, 0.72$ . This is not luck: the map is odd because $\nabla L (- θ) = - \nabla L (θ)$ for the bowl, so flipping the start's sign flips every $g_{t}$ , every $v_{t}$ , and every $θ_{t}$ in lock-step. Any symmetric (even) loss inherits this mirror property, so you never have to re-derive the left-hand side — just negate the right-hand answers. That symmetry is your free sanity check on any bowl-like problem. ✓

Look at figure s06: the magenta curve is the negative-start run and the violet dashed curve is Ex 1's positive-start run; the orange dotted horizontal is $θ = 0$ . Notice the two curves are perfect mirror images across that zero-line — each magenta point is exactly the reflection of its violet twin, which is the odd symmetry made visual.

Worked example Ex 8 — Cell H: real-world word problem

"Your training loss on a mini-batch stream (see Stochastic Gradient Descent (SGD)) reports the same average gradient $g = 0.5$ for many steps because you're on a long gentle plateau. You set $η = 0.02$ and $β = 0.9$ . In the steady state, how far does each parameter move per step, and how does that compare to plain SGD?"

Forecast: will momentum help on a flat plateau, or only in valleys?

Steady-state velocity: $v_{\infty} = \frac{g}{1 - β} = \frac{0.5}{0.1} = 5$ . Why this step? a constant gradient is exactly Cell E — the EWMA saturates at $g / (1 - β)$ .
Per-step move: $η v_{\infty} = 0.02 \times 5 = 0.1$ . Why this step? the actual displacement is $η$ times the velocity, not the raw gradient.
Plain SGD move: $η g = 0.02 \times 0.5 = 0.01$ . Why this step? SGD uses the raw gradient with no memory.

Verify (with units of "parameter distance per step"): momentum crawls at $0.1$ units/step, SGD at $0.01$ units/step, ratio $0.1/0.01 = 10 = \frac{1}{1 - β}$ . If the plateau is $50$ units long, momentum crosses it in $\approx 500$ steps versus SGD's $\approx 5000$ — a concrete $10 \times$ wall-clock saving on plateau-heavy problems, which is why they often prefer momentum or Adam Optimizer over bare SGD. ✓

Worked example Ex 9 — Cell I: exam twist, classical vs Nesterov side by side

"Starting from $θ_{0} = 1$ , $v_{0} = 0$ , $η = 0.1$ , $β = 0.9$ on $L = \frac{1}{2} θ^{2}$ , compute $θ_{2}$ for both methods and state which is nearer the minimum."

Forecast: which reaches lower loss after two steps?

Classical (from Ex 1): $θ_{2} = 0.72$ .
Nesterov (from Ex 3): $θ_{2} = 0.729$ . Why this step? Nesterov read the gentler look-ahead gradient $0.81$ , so it built less velocity and stayed slightly higher.
Loss compare: $L_{cl} = \frac{1}{2} (0.72)^{2} = 0.2592$ vs $L_{nes} = \frac{1}{2} (0.729)^{2} = 0.2657$ .

Verify (name the trap explicitly): on this smooth bowl over just two steps classical is marginally lower ( $0.2592 < 0.2657$ ), a difference of only $\approx 0.0065$ . The exam trap is concluding "classical is better" from this. Nesterov's real payoff is the anticipatory damping that prevents overshoot over many steps and gives the $O (1/ t^{2})$ convex rate — invisible in a 2-step run. Two data points cannot reveal an asymptotic rate. ✓

Recall Self-test: which cell is this?

A gradient stream reads $+ 2, - 2, + 2, - 2$ . Which matrix cell, and what happens to velocity? ::: Cell B — alternating; velocity stays small and bounded because opposite signs cancel. You set $β = 0$ . What does the update become? ::: Cell D3 — plain gradient descent, $θ_{t + 1} = θ_{t} - η g_{t}$ . Constant gradient $g = 0.3$ , $β = 0.8$ . Steady velocity? ::: $g / (1 - β) = 0.3/0.2 = 1.5$ (Cell E).