Exercises — Momentum and Nesterov momentum

The symbol is $\beta$, the momentum coefficient, with $\beta \in [0,1)$.

• If $\beta =0$: $v_t=g_t$ — no memory, we are back to plain gradient descent.
• As $\beta \to 1$: the sum $v_t={\sum }_k{\beta }^k{g}_{t-k}$ keeps more and more old gradients (each weight ${\beta }^k$ decays slowly), so the ball gets "heavier". $\eta$ scales the step size, not the memory — so the answer is $\beta$.

Recall $g_1=\nabla L({\theta }_0)$, the gradient at the starting point. $v_1=\beta v_0+g_1=0.9(0)+2=2$. Step $=\eta v_1=0.1\times 2=0.2$. Notice this is exactly the plain-GD step $\eta g_1=0.2$. Why? With no accumulated history ($v_0=0$), momentum has nothing to add on the first step. This is the "slow start" the parent note warns about — not a bug.

The gradient is just the current $\theta$, so $g_t={\theta }_t$. Keep every digit — small roundings compound.

• $v_1=0.9(0)+1=1$; ${\theta }_1=1-0.1(1)=0.9$.
• $v_2=0.9(1)+0.9=1.8$ (exactly $1.8$, not $1.79$); ${\theta }_2=0.9-0.1(1.8)=0.72$.
• $v_3=0.9(1.8)+0.72=2.34$; ${\theta }_3=0.72-0.1(2.34)=0.486$. Compare to plain GD (${\theta }_{t+1}=0.9{\theta }_t$): it would give $0.9,0.81,0.729$. Momentum reaches $0.486$ at step 3 — visibly ahead, because consistent-sign gradients keep adding into $v$.

Read the figure (s01): the amber momentum curve dives below the cyan plain-GD curve after step 2. That gap is the accumulated velocity $v$: because $g_t={\theta }_t$ never changes sign on the way down, each new gradient adds to $v$ instead of cancelling, so the step $\eta v$ keeps growing while plain GD's step $\eta {\theta }_t$ shrinks with $\theta$. The amber arrow marks exactly where momentum overtakes.

At the fixed point $v_{\infty }=\beta v_{\infty }+g$, so $v_{\infty }(1-\beta )=g$ and $v_{\infty }=\frac{g}{1-\beta }.$ The step is $\eta v_{\infty }=\frac{\eta }{1-\beta }g$, so ${\eta }_{\text{eff}}=\frac{\eta }{1-\beta }=\frac{0.1}{1-0.9}=\mathrm{1.0.}$ That is a 10× amplification of the raw learning rate along a consistent direction — the parent note's "10× speed-up".

• $v_1=0.9(0)+1=1$.
• $v_2=0.9(1)-1=-0.1$.
• $v_3=0.9(-0.1)+1=0.91$.
• $v_4=0.9(0.91)-1=-0.181$. The velocity stays small and bounded: for an alternating unit gradient the worst case is a geometric sum $\le {\sum }_k{\beta }^k=\frac{1}{1-\beta }=10$, but the sign flips keep it hovering near $\pm 1$ instead. Compare a consistent-sign coordinate where $v\to 10$. So momentum shrinks the oscillating direction relative to the consistent one — exactly the ill-conditioning cure.

Read the figure (s02): the cyan bars are the raw $\pm 1$ gradients; the amber bars are the velocity $v_t$. Watch the amber bars stay pinned near zero (about $\pm 1$ at most) while the dotted white line at $10$ marks where velocity would sit if the gradients all agreed. The visual message: the same $\beta$ that would give a $10\times$ boost gives almost nothing here — that selectivity is the whole point.

Plain GD gives ${\theta }_t=(1-\eta {)}^t{\theta }_0$. This shrinks to $0$ iff $|1-\eta |<1$, i.e. $0<\eta <2.$ Now imagine two coordinates with curvatures ${\lambda }_{\max }$ (steep) and ${\lambda }_{\min }$ (flat). The steep coordinate needs $\eta <2/{\lambda }_{\max }$ to not diverge; the flat one converges at rate $(1-\eta {\lambda }_{\min })$. Because $\eta$ is shared, the steep constraint caps $\eta$, making the flat direction crawl at rate $\approx 1-2{\lambda }_{\min }/{\lambda }_{\max }=1-2/\kappa$ where $\kappa ={\lambda }_{\max }/{\lambda }_{\min }$ is the condition number (see Hessian and Condition Number). Large $\kappa$ ⇒ painfully slow. Momentum fixes this without touching $\eta$.

Step 1: ${\tilde{\theta }}_1=1-0.1(0.9)(0)=1$. Gradient at look-ahead $=1$. $v_1=0.9(0)+1=1$; ${\theta }_1=1-0.1(1)=0.9$. (Identical to classical — $v_0=0$.) Step 2: ${\tilde{\theta }}_2=0.9-0.1(0.9)(1)=0.81$. Gradient at look-ahead $=0.81$ (smaller than the $0.9$ classical uses). $v_2=0.9(1)+0.81=1.71$; ${\theta }_2=0.9-0.1(1.71)=0.729$. Compare: classical (from L2.1, computed exactly) had $v_2=1.8,{\theta }_2=0.72$. Nesterov's $v_2=1.71$ is smaller because it evaluated the gradient at the anticipated, closer-to-minimum point $0.81$ rather than at $0.9$. It aims a touch gentler → less overshoot. On this simple 1-D bowl classical actually lands marginally closer this single step ($0.72$ vs $0.729$), but Nesterov's anticipatory damping pays off on oscillating/ill-conditioned surfaces where classical overshoots.

Derivation. From ${\theta }_{t+1}={\theta }_t-\eta v_t$ we get $v_t=({\theta }_t-{\theta }_{t+1})/\eta$, and likewise $v_{t-1}=({\theta }_{t-1}-{\theta }_t)/\eta$. Substitute into $v_t=\beta v_{t-1}+g_t$: $\frac{{\theta }_t-{\theta }_{t+1}}{\eta }=\beta \frac{{\theta }_{t-1}-{\theta }_t}{\eta }+g_t.$ Multiply by $-\eta$ and solve for ${\theta }_{t+1}$: ${\theta }_{t+1}={\theta }_t-\eta g_t+\beta ({\theta }_t-{\theta }_{t-1}).$ The term $\beta ({\theta }_t-{\theta }_{t-1})$ literally is $\beta$ times the previous step — "keep going the way you were going." Numeric check (fully exact, no rounding). From L2.1, ${\theta }_0=1,{\theta }_1=0.9$, and $g_1=\nabla L({\theta }_1)={\theta }_1=0.9$. Then ${\theta }_2={\theta }_1-\eta g_1+\beta ({\theta }_1-{\theta }_0)=0.9-0.1(0.9)+0.9(0.9-1).$ Compute each term: $0.1(0.9)=0.09$ and $0.9(0.9-1)=0.9(-0.1)=-0.09$, so ${\theta }_2=0.9-0.09-0.09=\mathrm{0.72.}$ This matches L2.1's ${\theta }_2=0.72$ exactly — there is no $0.001$ discrepancy once we avoid the earlier mis-rounding of $v_2$. The two formulations (velocity-based and position-based) are algebraically identical.

Use ${\eta }_{\text{eff}}=\frac{\eta }{1-\beta }$. Solve for $\beta$: $1-\beta =\frac{\eta }{{\eta }_{\text{eff}}}=\frac{0.05}{0.5}=0.1\Rightarrow \beta =\mathrm{0.9.}$ So $\beta =0.9$ gives a $10\times$ steady-state amplification, turning a stable $\eta =0.05$ into an effective $0.5$ only where gradients agree — precisely the direction-selective boost momentum is for. This is the design logic behind picking $\beta =0.9$ as a default.

(a) Unrolling each: $v_t={\sum }_{k=0}^{t-1}{\beta }^k{g}_{t-k},m_t=(1-{\beta }_1){\sum }_{k=0}^{t-1}{\beta }_1^k{g}_{t-k}.$ So $m_t=(1-{\beta }_1)v_t$ (with $\beta ={\beta }_1$). Since the update multiplies by $\eta$, using $m_t$ instead of $v_t$ just rescales the effective learning rate by $(1-{\beta }_1)$: ${\eta }_{\text{Adam}}{m}_t={\eta }_{\text{Adam}}(1-{\beta }_1)v_t$. Same direction, same trajectory shape — only $\eta$ is relabelled.

The why behind the normalization. The factor $(1-{\beta }_1)$ is chosen so the weights sum to one: for a long history the geometric series $(1-{\beta }_1){\sum }_k{\beta }_1^k=(1-{\beta }_1)\cdot \frac{1}{1-{\beta }_1}=1$. That makes $m_t$ a genuine weighted average of past gradients — its scale matches a single gradient, so it is interpretable and comparable across steps. The parent's $v_t$ is an unnormalized sum whose weights instead total $\frac{1}{1-\beta }$ ($=10$ for $\beta =0.9$); that is why $v_t$ silently carries the $10\times$ effective-LR boost inside it. Neither is "wrong" — they just push the same $\frac{1}{1-\beta }$ factor into different places (into $v$ for momentum, into $\eta$ for Adam).

(b) Because $m_t$ is a normalized average, at $t=1$ we have $m_1=(1-{\beta }_1)g_1$, which underestimates $g_1$ by the factor $(1-{\beta }_1)$ (e.g. $0.1\times$ for ${\beta }_1=0.9$): the average hasn't "filled up" its window yet, so it is diluted toward the zero we initialised with. Bias correction ${\hat{m}}_1=m_1/(1-{\beta }_1^1)=m_1/(1-{\beta }_1)=g_1$ exactly cancels that dilution. Plain momentum's $v_t$ is not normalized, so it never claims to be an average and simply "accepts a slow start" for the first few steps rather than correcting — that is the parent note's warm-up remark. See Exponentially Weighted Moving Average for the bias-correction algebra.