Momentum and Nesterov momentum: Question bank

Before we start, a quick shared vocabulary so no symbol is used un-earned:

True or false — justify

The true/false verdict is the easy part; the sentence after it is the whole point.

TF1. Momentum is just plain Gradient Descent with a larger learning rate

η

False — a larger

η

scales every direction including the steep oscillating one (→ divergence), whereas momentum only enlarges the consistent direction because averaging cancels the flipping one. It is direction-selective.

TF2. A larger

β

(say

0.999

) always converges faster than

β = 0.9

False — too much memory makes the ball so heavy it overshoots and orbits the minimum; velocity can't decelerate fast enough. There is a sweet spot, usually

0.9

–

0.99

TF3. The very first momentum step (

v_{0} = 0

) is identical to a plain gradient-descent step.

True — with no history,

v_{1} = β \cdot 0 + g_{1} = g_{1}

, so

θ_{1} = θ_{0} - η g_{1}

, exactly plain GD. The speed-up only appears once gradients start accumulating.

TF4. If a gradient stays constant at

g

forever, the momentum step size grows without bound.

False — the velocity converges to the geometric sum

g / (1 - β)

, a finite value; the effective learning rate saturates at

η / (1 - β)

, not infinity.

TF5. Nesterov and classical momentum differ only cosmetically in their equations.

False — evaluating the gradient at the look-ahead point

\tilde{θ}_{t}

adds a genuine anticipatory damping term to the dynamics, improving the convex convergence rate to

O (1/ t^{2})

from

O (1/ t)

TF6. Momentum helps most on a perfectly circular (isotropic) loss bowl.

False — a circular bowl has

κ = 1

, no zig-zag to cancel, so plain GD already goes straight in; momentum shines precisely when

κ ≫ 1

and the valley is a narrow canyon.

TF7. With

β = 0

, momentum reduces exactly to plain gradient descent.

True —

v_{t} = 0 \cdot v_{t - 1} + g_{t} = g_{t}

, so

θ_{t + 1} = θ_{t} - η g_{t}

. Momentum is a strict generalisation of GD.

TF8. Momentum can keep moving even at a point where the gradient is exactly zero.

True — at a saddle or flat spot

g_{t} = 0

but

v_{t} = β v_{t - 1}

is still nonzero, so the ball coasts through the flat region — a real advantage of inertia.

TF9. Momentum changes which minimum the loss surface has.

False — momentum only changes the path taken; the minima of

L

are properties of

L

alone. It affects how you arrive, not where.

TF10. In an oscillating direction, momentum's velocity blows up because it keeps adding gradients.

False — alternating

+ 1, - 1, + 1, \dots

gradients cancel in the EWMA, keeping

v

small and bounded, which is exactly why the parameter stops bouncing off the walls.

Spot the error

Each statement contains one wrong claim — name it.

SE1. "Because

v

averages gradients, momentum is unbiased from step one, so no warm-up is needed."

Error: early velocities are biased low — the geometric sum hasn't warmed up, so

v

underestimates the true averaged gradient. This is why Adam adds bias correction; plain momentum just accepts a slow start.

SE2. "The heavy-ball rewrite

θ_{t + 1} = θ_{t} - η g_{t} + β (θ_{t} - θ_{t - 1})

adds a brand new correction unrelated to the previous move."

Error: the term

β (θ_{t} - θ_{t - 1})

is literally the previous step, scaled — it means "keep going the way you were going", not something new.

SE3. "Nesterov computes the gradient at the current point

θ_{t}

, then adds an extra look-ahead term afterward."

Error: Nesterov evaluates the gradient at the look-ahead point

\tilde{θ}_{t} = θ_{t} - η β v_{t - 1}

; the anticipation is inside the gradient evaluation, not an add-on term.

SE4. "Since RMSProp and Adam adapt the learning rate per parameter, they already do what momentum does, so momentum is redundant."

Error: RMSProp adapts step magnitude per coordinate; momentum smooths the direction over time. Adam combines both — they solve different problems, not the same one.

SE5. "With

β = 0.9

the effective learning rate is

0.9 η

Error: the effective learning rate in a steady direction is

η / (1 - β) = η /0.1 = 10 η

, a ten-fold increase, not a reduction.

SE6. "Because momentum accumulates past gradients, using it with SGD mini-batches makes the noise worse."

Error: averaging past (noisy) gradients reduces variance — the EWMA smooths mini-batch noise, which is one reason momentum is standard with SGD.

Why questions

W1. Why do we step along the velocity

v_{t}

instead of the raw gradient

g_{t}

Because the raw gradient in an ill-conditioned valley keeps flipping sign and cancels its own progress; the velocity is a smoothed gradient that keeps signal (consistent direction) and kills noise (oscillation).

W2. Why does the oscillating component cancel while the consistent one adds?

In the EWMA, a component that keeps the same sign accumulates term-by-term, while one that flips sign produces near-cancelling terms — signal reinforces, noise self-destructs.

W3. Why does Nesterov "look ahead" before computing the gradient?

The ball is about to jump by

β v

anyway, so measuring the slope at the anticipated landing spot gives more up-to-date information and lets it brake before the curve instead of after.

W4. Why is the tool an exponentially weighted average rather than a plain average of all past gradients?

An exponential decay makes recent gradients count most and old, stale ones fade geometrically — it tracks a moving optimum without needing to store the full history.

W5. Why can't we cure ill-conditioning by simply shrinking

η

A tiny

η

stops divergence in the steep direction but makes progress in the flat direction painfully slow; momentum instead speeds the flat direction while cancelling the steep oscillation — decoupling the two.

W6. Why does the effective learning rate become

η / (1 - β)

and not something else?

Because a constant gradient

g

drives

v

to the geometric-series limit

g \sum_{k \geq 0} β^{k} = g / (1 - β)

; multiplying by

η

gives the sustained step size.

W7. Why is Nesterov's theoretical advantage stated as a rate (

O (1/ t^{2})

O (1/ t)

) rather than a constant factor?

The anticipatory damping changes how the error decays with iteration count, not just by a fixed multiplier — a fundamentally faster asymptotic shape on convex problems.

Edge cases

E1. What happens with

β = 0

You recover plain gradient descent exactly; velocity has no memory, so

v_{t} = g_{t}

E2. What happens as

β \to 1^{-}

The effective learning rate

η / (1 - β) \to \infty

and the ball becomes nearly frictionless — it will overshoot and orbit the minimum, oscillating for a long time. This is why

β = 1

is excluded.

E3. At a saddle point where

g_{t} = 0

but

v_{t - 1} \neq = 0

, what does momentum do?

It coasts through —

v_{t} = β v_{t - 1} \neq = 0

carries the parameters onward, helping escape flat saddle regions that stall plain GD.

E4. At the exact minimum where

g_{t} = 0

and the ball still has velocity, does it stop?

Not immediately — it overshoots past the minimum, then the gradient reverses and pulls it back; it settles only after the residual velocity decays, hence damped oscillation near the optimum.

E5. On a perfectly flat plateau (

g = 0

for many steps), how does velocity behave?

It decays geometrically as

v_{t} = β^{t} v_{0}

, so the ball keeps drifting but slows down — inertia carries it across the plateau before dying out.

E6. If gradients are pure alternating noise

+ g, - g, + g, \dots

with no true direction, where does momentum go?

Almost nowhere — the velocity stays small and bounded near zero, so the parameters barely move, which is the desired behaviour (don't chase noise).

E7. How should

β

interact with a changing learning rate from LR schedules?

A common practice is to warm up

β

(or

η

) early since velocity is under-estimated at the start; a heavy ball plus a large early

η

can destabilise before the average settles.

Recall One-line summary

Verdict is cheap; the justification is the skill — every trap here is really asking "signal accumulates, noise cancels, look-ahead brakes early."