Exercises — Learning rate effects on convergence

${\theta }_{t+1}={\theta }_t-\eta {\nabla }_{\theta }\mathcal{L}({\theta }_t)$

• ${\theta }_t$ ::: the parameters (weights) at step $t$ — the position on the loss surface.
• $\eta >0$ ::: the learning rate, a positive scalar that sets how far we move.
• ${\nabla }_{\theta }\mathcal{L}$ ::: the gradient, the arrow pointing uphill (steepest increase).
• the minus sign ::: flips uphill into downhill, so loss goes down.

Differentiate $w^2$: the slope of $w^2$ is $2w$, so $g(w)=\frac{d}{dw}{w}^2=2w.$ At $w=3$: $g(3)=2(3)=\boxed{6}$. The arrow has length $6$ and points in the $+w$ direction (uphill, away from the minimum at $0$).

$w_{t+1}=5-0.1\times 6=5-0.6=\boxed{4.4}.$ We moved from $5$ to $4.4$ — toward the minimum at $0$. The minus sign did its job: positive uphill arrow, so we stepped in the negative direction.

Gradient $g(w)=2w$.

• $w_1=1-0.3\cdot 2(1)=1-0.6=0.4$
• $w_2=0.4-0.3\cdot 2(0.4)=0.4-0.24=0.16$
• $w_3=0.16-0.3\cdot 2(0.16)=0.16-0.096=0.064$

Why a clean formula exists. Substitute the gradient into the update: $w_{t+1}=w_t-\eta \cdot 2w_t=(1-2\eta )w_t$. With $\eta =0.3$ this is $w_{t+1}=(1-0.6)w_t=0.4w_t$ — a linear recurrence, each term is $0.4$ times the one before. Unrolling from $w_0=1$: $w_1=0.4$, $w_2=0.4\cdot 0.4=0.4^2$, and in general $w_t=0.4^t$ (a one-line induction: if $w_t=0.4^t$ then $w_{t+1}=0.4\cdot 0.4^t=0.4^{t+1}$). That is clean geometric decay toward $0$.

Multiplier per step is $(1-1.5\cdot 2)=(1-3)=-2$.

• $w_1=1-1.5\cdot 2(1)=1-3=-2$
• $w_2=-2-1.5\cdot 2(-2)=-2+6=4$
• $w_3=4-1.5\cdot 2(4)=4-12=-8$

$|w_t|=2^t$: the value flips sign each step and doubles in magnitude. This is divergence — the step is so big we land farther out on the opposite wall each time. Figure s02 (below) plots these iterates climbing outward along the parabola.

Gradient: $g(w)=2(wx-y)x=2(w-2)$ (since $x=1$).

• $g(0)=2(0-2)=-4$, so $w_1=0-0.1(-4)=0.4$
• $g(0.4)=2(0.4-2)=-3.2$, so $w_2=0.4-0.1(-3.2)=0.72$

Moving up toward $w^{\ast }=2$. The multiplier here is $(1-0.1\cdot 2)=0.8$ acting on the distance to $w^{\ast }$: distance $2\to 1.6\to 1.28$.

One step multiplies $w$ by $r=(1-2\eta )$. Repeated multiplication stays bounded exactly when $|r|\le 1$, i.e. when $r$ lies between $-1$ and $1$: $-1\le 1-2\eta \le 1.$ Solve the two halves separately.

• Left inequality $-1\le 1-2\eta$: subtract $1$ from both sides $\Rightarrow -2\le -2\eta$; divide by $-2$ (flip the sign because we divide by a negative) $\Rightarrow \eta \le 1$.
• Right inequality $1-2\eta \le 1$: subtract $1$ $\Rightarrow -2\eta \le 0$; divide by $-2$ (flip again) $\Rightarrow \eta \ge 0$.

Combining: $0\le \eta \le 1$. So the stability ceiling is ${\eta }_{\max }=\frac{2}{\lambda }=\frac{2}{2}=\boxed{1}$ (at exactly $\eta =1$, $r=-1$, so it oscillates $\pm 1$ forever without shrinking). The one-step rate needs $r=0$: $1-2\eta =0\Rightarrow \eta =\frac{1}{2}=\boxed{0.5}$. This is Newton's step $1/\lambda$.

The condition is $\eta <\frac{2}{L}$ (see Lipschitz Continuity).

• Ceiling: $\eta <\frac{2}{8}=0.25$.
• Recommended safe rate: $\eta \approx \frac{1}{L}=\frac{1}{8}=\boxed{0.125}$ — half the ceiling, a comfortable margin.

(a) Every axis must stay stable, so the tightest one wins: $\eta <\frac{2}{{\lambda }_{\max }}=\frac{2}{50}=\boxed{0.04}.$ (b) $\kappa =\frac{{\lambda }_{\max }}{{\lambda }_{\min }}=\frac{50}{2}=\boxed{25}$. With $\eta$ capped by the steep $w_2$ axis, the shrink factor on the shallow $w_1$ axis is only $1-\eta {\lambda }_{\min }=1-0.04\cdot 2=0.92$ per step — painfully slow. The valley is long and narrow: the steep direction limits $\eta$, the shallow direction limits speed. This is exactly the pain the Hessian Matrix warns about and what Momentum and the Adam Optimizer were built to relieve. Figure s03 (below) shows the zig-zag path this forces.

Gradient $g=2cw$, so the multiplier is $r=1-2c\eta$. $0.2=1-2c(0.2)=1-0.4c\Rightarrow 0.4c=0.8\Rightarrow c=\boxed{2}.$ Check: $\lambda =2c=4$, ceiling $2/\lambda =0.5$, and $0.2<0.5$ — consistent with the clean decay we observed.

Stability: pick $\eta =0.04$ (the ceiling from Ex 3.3). Along $w_2$, multiplier $r=1-\eta {\lambda }_{\max }=1-0.04\cdot 50=-1$ — that oscillates forever, no shrink. Too aggressive. Back off to the safe $\eta =1/{\lambda }_{\max }=0.02$: $r=1-0.02\cdot 50=0\Rightarrow \text{w2 hits 0 in one step.}$ But then the $w_1$ axis shrink factor is $1-0.02\cdot 2=0.96$ — very slow. Two-phase idea: Phase 1 use $\eta =0.02$ (kills $w_2$ immediately); Phase 2, once $w_2\approx 0$, raise to $\eta =0.5$ (the one-step rate for ${\lambda }_{\min }=2$) to finish $w_1$ fast. Why Phase 2 may ignore the $w_2$ axis — the assumption made explicit. Because these axes are decoupled (the loss splits as $w_1^2+25w_2^2$, no cross-term), the $w_2$ update only depends on $w_2$. Once $w_2$ is exactly $0$, its gradient $50w_2=0$, so every later step leaves $w_2$ at $0$ regardless of $\eta$. That is the implicit assumption: Phase 2 is safe on the steep axis only because $w_2$ is already exactly zero, so the large ${\lambda }_{\max }$ curvature is never activated. In real (coupled, noisy) problems $w_2$ is only approximately $0$, so a large $\eta$ could re-excite it — which is why practitioners decay $\eta$ smoothly instead of jumping. If instead we insist on a single $\eta =0.02$: the $w_1$ axis needs $0.96^n\le 0.01$. Solve: $n\ge \frac{\ln 0.01}{\ln 0.96}=\frac{-4.605}{-0.0408}\approx 112.8\Rightarrow \boxed{n=113}.$ This is the whole reason Learning Rate Schedules and Adam Optimizer exist — one global $\eta$ cannot serve two curvatures at once.

Setup. $L$-smoothness means the slope ${\mathcal{L}}^{\prime }$ changes by at most $L$ per unit of $w$; the standard consequence (the "descent lemma") is $\mathcal{L}(w+\Delta )\le \mathcal{L}(w)+{\mathcal{L}}^{\prime }(w)\Delta +\frac{L}{2}{\Delta }^2.$ Plug in the step $\Delta =-\eta g$ (and ${\mathcal{L}}^{\prime }(w)=g$): $\mathcal{L}(w-\eta g)\le \mathcal{L}(w)+g(-\eta g)+\frac{L}{2}(\eta g{)}^2=\mathcal{L}(w)-\eta g^2+\frac{L}{2}{\eta }^2{g}^2.$ Guaranteed decrease is $D(\eta )=\eta g^2-\frac{L}{2}{\eta }^2{g}^2$. Maximise: set $\frac{dD}{d\eta }=g^2-L\eta g^2=0\Rightarrow \boxed{\eta =1/L}$. The best drop is $D(1/L)=\frac{g^2}{2L}$. And decrease is positive only while $\eta g^2>\frac{L}{2}{\eta }^2{g}^2$, i.e. $\eta <2/L$ — recovering the stability ceiling. This is why the parent note recommends $\eta \approx 1/L$: it is provably the best worst-case step.

Gradient $g=4w^3$; second derivative (curvature) ${\mathcal{L}}^{\prime \prime }=12w^2$. The curvature is not constant — it grows like $w^2$. Far out (large $|w_0|$) curvature is huge, so any fixed $\eta$ eventually violates the local ceiling $\eta <2/{\mathcal{L}}^{\prime \prime }=1/(6w^2)$ and overshoots. Concretely with $w_0=2,\eta =0.1$: $g=4\cdot 8=32$, $w_1=2-0.1\cdot 32=-1.2$. Note the sign flipped (we jumped across the minimum at $0$ to the negative side) even though the magnitude did shrink from $2$ to $1.2$. That sign-flip is the warning sign of overshoot: the step was large enough to cross the minimum. Push $\eta$ or $w_0$ a little higher and the flipped point lands farther out than it started — genuine divergence. Try $w_0=2,\eta =0.2$: $w_1=2-0.2\cdot 32=-4.4$, now $|w_1|=4.4>2$ — magnitude actually grows. Near $0$ the curvature $\to 0$, so the same $\eta$ becomes far too timid and progress stalls (steps shrink like $w^3$). Moral: no single constant $\eta$ fits a surface whose curvature changes by orders of magnitude — early steps need small $\eta$, late steps need larger effective $\eta$. This is precisely what Learning Rate Schedules (warm-up then decay) and per-parameter methods like Adam Optimizer adapt to automatically.