Visual walkthrough — Learning rate effects on convergence

---

Step 1 — What is a "loss landscape"?

WHAT. Imagine you have one knob to tune — call it $w$ (a single number, our parameter). For every setting of that knob, your model is a bit right or a bit wrong. We measure "how wrong" with a number called the loss, written $\mathcal{L}(w)$. Plot the knob on the flat ground and the loss as height, and you get a valley.

WHY. Training means finding the bottom of the valley — the knob setting where you are least wrong. Everything else on this page is about how to walk downhill without falling over.

PICTURE. The curve below is $\mathcal{L}(w)=w^2$, the simplest possible valley. The lowest point sits at $w=0$; that is the target we call $w^{\star }$ ("w-star", the best setting).

---

Step 2 — The slope tells us which way is downhill

WHAT. At any point on the curve, ask: if I nudge $w$ a tiny bit to the right, does the height go up or down, and how steeply? That steepness is the slope, and for a curve we compute it with the derivative, written $\nabla \mathcal{L}$ (read "grad L"). For $\mathcal{L}(w)=w^2$ the slope is $\nabla \mathcal{L}=2w$.

WHY a derivative and not something else? We need a local answer — which way is down right here, without seeing the whole valley. The derivative is exactly the tool that reports the local tilt of a curve. No other tool gives "steepness at a point."

$\underset{\text{the slope here}}{\underbrace{\nabla \mathcal{L}(w)}}=\underset{\text{formula for }{w}^2}{\underbrace{2w}}$

PICTURE. On the right wall ($w>0$) the slope is positive — the curve rises as you go right, so the arrow of steepest increase points right. On the left wall ($w<0$) the slope is negative — increase points left. Downhill is always the opposite of the slope arrow.

---

Step 3 — Predict the loss ahead with a straight-line guess

WHAT. First, one new symbol. When we move the knob from $w$ to a nearby setting, we write the change in $w$ as $\Delta w$ (read "delta w") — the capital Greek letter $\Delta$ just means "the amount by which something changed." So $\Delta w$ is literally how far we step, and the new knob-setting is $w+\Delta w$.

Standing at $w$, before we actually move, we guess what the loss will be after a step of size $\Delta w$. The cheapest honest guess is the first-order Taylor approximation: pretend the curve is a straight line with the slope we just measured.

$\mathcal{L}(w+\Delta w)\approx \underset{\text{height now}}{\underbrace{\mathcal{L}(w)}}+\underset{\text{slope}}{\underbrace{\nabla \mathcal{L}(w)}}\cdot \underset{\text{how far we step}}{\underbrace{\Delta w}}$

WHY this tool? We want to choose our step $\Delta w$ intelligently, which means predicting its effect before committing. Taylor's straight-line rule is the simplest predictor that uses the slope. It is only trustworthy for small $\Delta w$, and remembering that limitation is the whole story of the learning rate.

PICTURE. The horizontal gap $\Delta w$ is marked on the axis. The dashed line is the straight-line prediction; it hugs the true curve near $w$ and drifts away as $\Delta w$ grows. That drift is the danger zone.

---

Step 4 — Choosing the step: enter the learning rate $\eta$

WHAT. Look again at the prediction from Step 3, reading it as a function of our choice $\Delta w$:

$\mathcal{L}(w+\Delta w)\approx \mathcal{L}(w)+\nabla \mathcal{L}(w)\cdot \Delta w.$

We want this predicted loss to drop as much as possible, so we want the term $\nabla \mathcal{L}(w)\cdot \Delta w$ to be as negative as possible.

First, a trap to notice. Read that term greedily: if $\Delta w$ is a free number with no limit, we could just take $\Delta w$ enormous in the downhill direction and the straight-line prediction would promise a loss of $-\infty$. That is nonsense — the prediction is only a local straight-line guess (Step 3), trustworthy for small steps and wildly wrong far away. So the linear formula alone has no minimum; the greedy answer runs off to infinity.

The fix: cap the step length. The honest thing is to admit "I only trust this guess within a small radius," i.e. impose a size limit $|\Delta w|\le \eta \Vert \nabla \mathcal{L}(w)\Vert ,$ some allowed budget on how far we move. Now the problem is well-posed: among all steps no longer than the budget, which one drops the linear prediction most? Two numbers multiplied are most negative when they point exactly opposite, so the best direction is straight against the gradient, and we should spend the whole budget (go to the boundary). Pointing any other way wastes part of the step sideways.

That forces the update to be a fixed negative multiple of the gradient; call that positive multiplier $\eta$ (read "eta"), the learning rate — it is exactly the budget size we allowed ourselves:

$\Delta w=-\underset{\text{step budget}}{\underbrace{\eta }}\underset{\text{slope (direction)}}{\underbrace{\nabla \mathcal{L}(w)}}$

Plug into the update and you get the rule from the parent note:

$\boxed{w_{t+1}=\underset{\text{where we are}}{\underbrace{w_t}}-\underset{\text{how far}}{\underbrace{\eta }}\underset{\text{which way (times }-1)}{\underbrace{\nabla \mathcal{L}(w_t)}}}$

WHY split direction and distance? The minus sign is derived, not guessed: it is the only direction that makes the linear prediction go down fastest. The distance is not derivable from the linear guess at all — greedily it wants infinity — so we must impose it as the trust budget $\eta$. Direction comes from calculus; distance is our cap.

PICTURE. Same valley, same starting point, three different $\eta$: a timid step, a good step, and a reckless step. Watch where each lands.

---

Step 5 — Why "too big" isn't just slow, it's fatal

WHAT. Track what actually happens for $\mathcal{L}(w)=w^2$ (so $\nabla \mathcal{L}=2w$). The update becomes

$w_{t+1}=w_t-\eta (2w_t)=\underset{\text{shrink factor }r}{\underbrace{(1-2\eta )}}{w}_t.$

Every step multiplies your position by the same number $r=1-2\eta$.

WHY this matters. Repeating multiplication by $r$ gives $w_t=r^t{w}_0$. The fate of training is decided entirely by the size of $r$:

| $\eta$ | $r=1-2\eta$ | $|r|$ | behaviour | |---|---|---|---| | $0.1$ | $0.8$ | $=0.8<1$ | shrinks, same side — slow, steady | | $0.5$ | $0$ | $=0<1$ | one jump to zero (best case) | | $0.9$ | $-0.8$ | $=0.8<1$ | flips side each step, still shrinks | | $1.0$ | $-1$ | $=1$ boundary | flips forever, never settles | | $1.5$ | $-2$ | $=2>1$ | flips and grows — explodes |

PICTURE. The three trajectories: $|r|<1$ spirals inward to $0$, $|r|=1$ orbits forever, $|r|>1$ marches outward. This is the whole "small / good / large" story in one geometric fact.

---

Step 6 — The degenerate and edge cases

WHAT & PICTURE. We must cover every scenario so no reader is ambushed:

• Flat spot, $\nabla \mathcal{L}=0$. At the exact bottom the slope is zero, so $\Delta w=0$: you stay put no matter what $\eta$ is. Good — you have arrived. (Same thing traps you on a saddle or plateau; that is why Momentum exists.)
• The knife-edge $\eta =1/\beta$ (here $\eta =0.5$): $r=0$, one perfect leap to the minimum. This is Gradient Descent accidentally becoming Newton's method — only possible because a parabola has constant curvature.
• The oscillation boundary $\eta =2/\beta$ (here $\eta =1$): $r=-1$, you ping-pong between $+w_0$ and $-w_0$ forever. Loss never decreases, never explodes — pure limbo.
• Beyond, $\eta >2/\beta$: $|r|>1$, each step lands farther out. Parameters $\to \pm \infty$; you see NaN.

---

The one-picture summary

Everything collapses to one line: the multiplier $r=1-\eta \beta$. Its distance from zero is your convergence speed; the moment $|r|$ touches $1$ you stop converging; past $1$ you diverge.

Recall Feynman retelling — say it like you'd explain to a friend

You're standing on the side of a valley whose height is your "wrongness." You can feel the slope under your feet — that's the derivative — and it tells you which way is downhill. But the slope doesn't tell you how far the bottom is, so you pick a step size $\eta$ yourself. Take a step of size $\eta$ times the slope, downhill.

Here's the catch. For a simple bowl, every step just multiplies your distance-from-bottom by a fixed number $r=1-\eta \beta$, where $\beta$ is how sharply the bowl curves. If that number is between $-1$ and $1$, each step shrinks the distance and you glide in. If $\eta$ is small, $r$ is near $1$ and you crawl. Right in the sweet spot, $r$ is near $0$ and you drop almost instantly. But if $\eta$ is too big, $r$ becomes bigger than $1$ in size — now every step grows the distance, you leap past the bottom onto the far wall, higher up, then even higher, and blow up to infinity. The whole learning-rate story is just: keep $|1-\eta \beta |$ under $1$.

Recall Quick self-check

For $\mathcal{L}=w^2$ with $w_0=1,\eta =0.3$, what is $w_1$? ::: $w_1=(1-2\cdot 0.3)\cdot 1=0.4$. What is the shrink factor $r$ for $\eta =0.1$? ::: $r=1-2(0.1)=0.8$. Above what $\eta$ does $w^2$ diverge? ::: $\eta >2/\beta =2/2=1$.

Related tools that fix the pain shown here: Learning Rate Schedules (shrink $\eta$ over time), Adam Optimizer (per-parameter $\eta$), Condition Number and the Hessian Matrix (where $\beta$ comes from in many dimensions), Lipschitz Continuity (the precise meaning of "curvature bound $\beta$"), and Backpropagation (how $\nabla \mathcal{L}$ is computed in the first place).