Foundations — Momentum and Nesterov momentum

This is the foundations page for Momentum and Nesterov momentum. The parent note throws a lot of symbols at you at once — $\theta$, $\nabla$, $\eta$, $v$, $\beta$, $\kappa$, and an "exponentially weighted average". Here we earn every single one from zero, in an order where each piece leans on the one before it. Nothing below assumes you have seen calculus or vectors before.

0. The playground: a loss landscape

Before any symbol, picture the thing we are moving on.

Everything in this topic is just clever ways of walking to that red dot without falling over.

1. $\theta$ — the parameters (the knobs)

• Plain words: the list of every number we are allowed to change.
• The picture: your horizontal location on the hill in figure s01. Move $\theta$ → you slide sideways to a new spot with a new height.
• Why the topic needs it: momentum is a rule for changing $\theta$ step by step. Without a symbol for "where we are", we can't describe "where we go next".

We write ${\theta }_t$ with a little subscript $t$ = "the position at step number $t$". So ${\theta }_0$ is the start, ${\theta }_1$ after one step, and so on. $t$ is just a counter, like frame numbers in a flip-book.

2. $L(\theta )$ — the loss (the height)

• The picture: the height of the terrain directly above position $\theta$ in s01.
• Why: we can't aim for "lowest" unless we have a number that says how low we are. $L$ is that number.

The whole game: make $L(\theta )$ as small as possible by choosing $\theta$ well. This links to Gradient Descent, the base method we're improving.

3. Slope, and why we need calculus: ${\nabla }_{\theta }L$

Here a mathematical tool enters, so we say why this tool and not another.

• In 1-D (one knob) it's just the ordinary slope: positive slope = ground rises to the right, negative = rises to the left.
• In many knobs it's an arrow made of one slope per knob, packed together — pointing up the steepest way.
• The picture: the orange arrow in s02 points uphill. To go down, we walk the opposite way, $-{\nabla }_{\theta }L$.
• Why the topic needs it: every update rule in this topic starts from ${\nabla }_{\theta }L$ — it's the "which way is down" signal. Momentum will average several of these arrows instead of trusting just one.

4. $\eta$ — the learning rate (step length)

• The picture: the length of one footstep. Big $\eta$ = long stride (risk overshooting), small $\eta$ = tiny careful step (slow).
• Why: the gradient gives a direction; $\eta$ turns it into an actual move. Together they make the basic rule below. Choosing $\eta$ over time is its own subject — see Learning Rate Schedules.

Now we can read the parent's plain gradient descent rule with zero mystery: ${\theta }_{t+1}={\theta }_t-\eta {\nabla }_{\theta }L({\theta }_t)$

5. The zig-zag problem: curvature, the Hessian $H={\nabla }^{2}L$, and $\kappa$

Why do we even need momentum? Because plain steps misbehave in a narrow valley.

• Why the topic needs it: $\kappa$ is the precise name for "ill-conditioned". Large $\kappa$ is the disease; momentum is the cure. This is the reason the whole topic exists.

6. $v$ and $\beta$ — velocity and memory (the rolling ball)

Now the two starring symbols. First the idea, then the symbols.

• The picture: $\beta$ is the ball's heaviness. Heavy ball (big $\beta$) barrels through little bumps but is slow to turn.

7. The tool behind $v$: the Exponentially Weighted Moving Average

The velocity update hides one more tool. Name it, motivate it.

Here we make the sign bookkeeping explicit, matching the parent note's convention. The parent accumulates the raw (uphill) gradient into the velocity and puts the minus sign in the position step, so the two together still move downhill:

Unroll the first line (using $v_0=0$) and each past gradient carries weight ${\beta }^k$ — recent = heavy, old = faint. This is exactly the machinery of Exponentially Weighted Moving Average, reused inside Adam Optimizer and RMSProp too.

• Why the topic needs it: these two lines are classical momentum. Understanding the EWMA (line 1) plus where $\eta$ and the minus sign go (line 2) = understanding why consistent directions add and oscillating ones cancel.

8. $\tilde{\theta }$ — the look-ahead point (Nesterov's twist)

• The picture: before committing, peek one step ahead and read the slope at the peek-spot. If it warns "you're about to overshoot", correct early.
• Why: this single substitution is the only difference between classical and Nesterov momentum in the parent note. The position step stays the same, ${\theta }_{t+1}={\theta }_t-\eta v_t$; only where the gradient is measured changes.

Prerequisite map

Equipment checklist

Cover the right side and test yourself. If any answer surprises you, reread that section.