Visual walkthrough — Momentum and Nesterov momentum

We build on the idea of plain gradient descent and the smoothing trick of an exponentially weighted moving average. Everything else we grow from scratch.

---

Step 1 — The hill, the ball, and the arrow that points downhill

WHAT. Picture a landscape. The height at each spot is the loss — call it $L$. Low ground = good (small loss). We stand at a location described by a number (or a list of numbers) called $\theta$ — the parameters of our model.

WHY. Before any update rule, we need a single honest arrow that says "downhill is that way." That arrow is the gradient, written ${\nabla }_{\theta }L(\theta )$. Read it as: "the direction of steepest increase of the loss." So its opposite, $-{\nabla }_{\theta }L(\theta )$, points downhill.

PICTURE. In the figure the gray curves are contour lines (lines of equal height, like a topographic map). The blue arrow at the ball is $-{\nabla }_{\theta }L(\theta )$: the steepest-descent direction, perpendicular to the contour it sits on.

The plain-GD rule is just "step downhill by $\eta$": ${\theta }_{t+1}=\underset{\text{where I am}}{\underbrace{{\theta }_t}}-\underset{\text{stride length}}{\underbrace{\eta }}\underset{\text{uphill arrow (so minus = downhill)}}{\underbrace{{\nabla }_{\theta }L({\theta }_t)}}$

The subscript $t$ is just a step counter: ${\theta }_0$ is the start, ${\theta }_1$ after one step, and so on.

---

Step 2 — Why one arrow is not enough: the zig-zag valley

WHAT. Squash the bowl so it is a narrow valley: very steep across, very gentle along. This is the ill-conditioned case the parent note warns about.

WHY. In a narrow valley the downhill arrow mostly points across the valley (into the steep wall), only a little along it (toward the true minimum). Plain GD must keep $\eta$ tiny or it launches up the opposite wall — so it zig-zags and crawls.

PICTURE. The orange path bounces wall-to-wall. Notice the key fact: the across-valley part of the arrow keeps flipping sign every step (left, right, left, right), while the along-valley part always points the same way (toward the minimum).

This is exactly the pathology that also motivates RMSProp and Adam Optimizer — but momentum fixes it the simplest way: by remembering.

---

Step 3 — Memory as a running average: the velocity $v$

WHAT. Keep a running, fading memory of the arrows we have seen. Call it the velocity $v$. We start with no memory at all, $v_0=0$ (an empty running total), and each step we forget a bit of the old velocity and add the new gradient.

WHY. "Forget a fraction, add the new" is precisely an exponentially weighted moving average. The consistent (along-valley) direction survives because its contributions all point the same way; the flipping (across-valley) direction shrinks because plus and minus cancel inside the sum. That is the "add up vs cancel" wish of Step 2, made into an equation.

$v_t=\underset{\begin{array}{c}\text{how much}\\ \text{past to keep}\end{array}}{\underbrace{\beta }}\underset{\text{old memory}}{\underbrace{v_{t-1}}}+\underset{\text{fresh arrow}}{\underbrace{{\nabla }_{\theta }L({\theta }_t)}},\underset{\text{empty memory at start}}{\underbrace{v_0=0}}$

Here $\beta \in [0,1)$ (beta) is the momentum coefficient — physically, how heavy the ball is. $\beta =0$ means no memory (back to plain GD); $\beta$ near $1$ means a long memory.

PICTURE. The stacked, fading arrows show the unrolled sum. The freshest arrow is full-strength; each older one is dimmer by a factor $\beta$.

---

Step 4 — Step along the memory, not the raw arrow

WHAT. Replace the raw gradient in the plain-GD rule with the velocity $v_t$.

WHY. We already argued $v_t$ is the good direction (signal amplified, noise cancelled), and $v_t$ points uphill, so $-v_t$ is downhill. Walk along $-v_t$.

${\theta }_{t+1}=\underset{\text{here}}{\underbrace{{\theta }_t}}-\underset{\text{stride}}{\underbrace{\eta }}\underset{\begin{array}{c}\text{smoothed uphill dir.}\\ -v_t\text{ is downhill}\end{array}}{\underbrace{v_t}}$

PICTURE. Two paths on the same valley: gray = plain GD (still zig-zagging), green = momentum (smooth, hugging the valley floor). The green ball's step is longer along the valley and shorter across it — exactly the selectivity we wanted.

---

Step 5 — How big does the "consistent" speed-up get?

WHAT. Ask: if the gradient were a constant forever (a perfectly steady slope), what velocity do we settle at? Give that constant gradient a name: let $g:={\nabla }_{\theta }L(\theta )\text{held fixed (the same uphill arrow every step).}$ So $g$ is nothing new — it is just the gradient ${\nabla }_{\theta }L(\theta )$ in the special case where it never changes, written as a short symbol so the algebra is readable.

WHY. This measures the payoff of memory in the best case — a purely consistent direction.

Starting from $v_0=0$, if ${\nabla }_{\theta }L(\theta )=g$ every step, the recursion $v_t=\beta v_{t-1}+g$ climbs toward a steady value $v_{\infty }$ satisfying $v_{\infty }=\beta v_{\infty }+g$, so $v_{\infty }=\frac{g}{1-\beta }⟹\text{effective stride}=\frac{\eta }{1-\beta }.$ Term-by-term: the $\frac{1}{1-\beta }$ is a geometric-series gain. With $\beta =0.9$ that gain is $\frac{1}{1-0.9}=10$ — a 10× acceleration along steady directions.

PICTURE. A curve of $v$ climbing from $v_0=0$ and levelling off at $g/(1-\beta )$, with the dashed asymptote marked for $\beta =0.9$.

---

Step 6 — The degenerate case: pure oscillation

WHAT. Take a coordinate whose gradient alternates perfectly: $+1,-1,+1,-1,\dots$ (crossing a symmetric valley). Watch $v$ with $\beta =0.9$, again starting from $v_0=0$.

WHY. This is the worst input for plain GD and the case momentum is built to tame — the edge case must be shown, not assumed.

$v_1=1,v_2=0.9-1=-0.1,v_3=0.9(-0.1)+1=0.91,v_4=0.9(0.91)-1=-0.181,\dots$

PICTURE. The raw gradient (orange spikes $\pm 1$) versus the velocity (green, hugging zero). The green curve stays small and bounded — the wild swings largely cancel, so the parameter barely wiggles.

---

Step 7 — Nesterov: look before you leap

WHAT. Classical momentum measures the arrow where it stands, then takes its full step ${\theta }_{t+1}={\theta }_t-\eta v_t$. Nesterov first peeks at the spot that the momentum part of that step would carry it to, and measures the arrow there.

WHY. Before committing, the ball is going to move; the momentum-only part of the classical step displaces ${\theta }_t$ by $-\eta \beta v_{t-1}$ (that is, $\eta$ times $\beta v_{t-1}$, the carried-over memory). Measuring the slope at the current spot is stale; measuring it at that anticipated spot gives fresher information, so momentum can brake before the curve instead of after it.

$\underset{\text{(1) peek where the momentum part carries me}}{\underbrace{{\tilde{\theta }}_t={\theta }_t-\eta \beta v_{t-1}}}$ $\underset{\text{(2) update memory using the arrow AT the peek}}{\underbrace{v_t=\beta v_{t-1}+{\nabla }_{\theta }L({\tilde{\theta }}_t)}}$ $\underset{\text{(3) now actually jump}}{\underbrace{{\theta }_{t+1}={\theta }_t-\eta v_t}}$

Term-by-term: ${\tilde{\theta }}_t$ (theta-tilde) is the look-ahead point; the only change from classical is that the gradient is read at ${\tilde{\theta }}_t$ instead of at ${\theta }_t$.

PICTURE. From the ball, a faded arrow shows the momentum-only peek to ${\tilde{\theta }}_t$; the red arrow is the gradient measured there; the green arrow is the resulting corrected step, which lands nearer the minimum than the classical step (shown gray).

---

The one-picture summary

Everything above compressed: the narrow valley, plain GD's gray zig-zag, momentum's smooth green sweep, and Nesterov's peek-then-jump correction — all on one map.

Recall Feynman retelling — the whole walkthrough in plain words

We stood on a hill and drew one arrow pointing downhill (Step 1). In a narrow valley that single arrow was useless — it made us bounce off the walls while barely crawling toward the bottom (Step 2). So we gave the ball a memory, starting empty ($v_0=0$): keep a fading average of every uphill arrow we've seen (Step 3), and walk along the opposite of that memory, $-v_t$, instead of the newest arrow (Step 4). Because arrows that agree pile up and arrows that flip cancel, we sped up along the valley by up to ten times (Step 5) while our sideways bouncing shrank to almost nothing (Step 6). Finally, we made a smarter ball that first peeks where its momentum is about to fling it, checks the slope there, and brakes early if it's about to overshoot (Step 7). That peeking ball is Nesterov, and it reaches the bottom fastest and smoothest of all.

---

Quick self-check