Gradient descent intuition and update rule: Visual walkthrough

This page rebuilds the single most important line in the parent note from nothing. No slopes, no dot products, no gradients assumed. Every symbol earns its place before it appears, and every step comes with a picture that carries the idea.

Because we are building from zero, we first fix three tiny pieces of shorthand that will appear throughout, so nothing is ever a surprise:

The sign function is worth seeing, since we lean on it in Step 3. Its picture is a flat staircase with a single jump: stuck at $- 1$ for all negative inputs, a lone dot at $0$ when the input is exactly zero, and stuck at $+ 1$ for all positive inputs. Notice it is completely blind to size — an input of $2$ and an input of $2000$ both come out $+ 1$ .

Figure — Gradient descent intuition and update rule

With those in hand, here is the line we are going to earn, piece by piece:

$θ_{t + 1} = θ_{t} - η \frac{d L}{d θ}_{θ_{t}}$

Every symbol here — $L$ , the slope $\frac{d L}{d θ}$ , the learning rate $η$ — is defined in the steps below before we lean on it. When $θ$ has many knobs at once, that single slope becomes an object called the gradient, written $\nabla L (θ_{t})$ ; we introduce it properly at the end of Step 5. Until then we stay with one knob and never write $\nabla$ .

See the parent topic for the big-picture analogy; here we do the pictures.

Step 1 — What is a "loss", and what does its picture look like?

WHAT. Picture one single knob $θ$ you can slide left or right. For every position of the knob, there is one "wrongness" number. Plot knob-position along the flat bottom, wrongness going up. You get a curve — a landscape.

WHY a curve and not a formula-blob? Because a picture lets us see the goal: the lowest point of the curve is the knob-setting that makes us least wrong. That lowest point has a name.

PICTURE. The bowl-shaped curve below. The horizontal axis is $θ$ (the knob). The vertical axis is $L$ (wrongness). The pink dot at the bottom is $θ^{*}$ , the answer we are hunting.

The catch: in real problems we cannot see the whole curve. We only ever stand at one spot and get to feel the ground right under our feet. So the rest of this page is: how do you find the bottom of a valley you cannot see, feeling only your immediate surroundings?

Step 2 — "Feeling the ground": the slope

WHAT. At a point $θ$ , draw the straight line that just grazes the curve there — the line that matches the curve's tilt exactly at that spot. That grazing line is the tangent line. Its steepness is the slope at $θ$ .

WHY a tangent line and not the curve itself? Because a curve is complicated, but a straight line is simple: a straight line has exactly one steepness number. Right where we stand, the curve and its tangent line agree, so the line's steepness is a faithful local report of the ground.

The tool that gives us this number is the derivative.

PICTURE. Two spots on the same bowl. On the right wall the tangent line climbs → positive slope. On the left wall the tangent line falls → negative slope. At the bottom the tangent is flat → zero slope.

Read the signs off the picture and a plan appears: walk in the direction that makes wrongness go down. Which direction is that? That is exactly Step 3.

Step 3 — The sign of the slope tells you which way to step

WHAT. We want to move the knob so $L$ decreases. Recall $sign (x)$ from the opening: it reports only which way a number leans ( $+ 1$ , $- 1$ , or $0$ ). Apply it to the slope and reason out both walls of the bowl.

Case A — we are on the right wall (slope positive, $sign (\frac{d L}{d θ}) = + 1$ ). Positive slope means "right = uphill", so downhill is to the left. To decrease $L$ , move $θ$ left (make it smaller).

Case B — we are on the left wall (slope negative, $sign (\frac{d L}{d θ}) = - 1$ ). Negative slope means "right = downhill", so downhill is to the right. To decrease $L$ , move $θ$ right (make it larger).

WHY the magic that follows works. Notice the beautiful pattern:

slope positive → step in the negative direction (left),
slope negative → step in the positive direction (right).

In both cases, the step direction is the opposite sign of the slope. That single sentence is the seed of the whole update rule.

PICTURE. The same bowl with a blue arrow on each wall. On the right wall (slope $+$ ) the arrow points left. On the left wall (slope $-$ ) the arrow points right. Both arrows point toward the bottom.

Step 4 — Fold direction AND distance into one number: $- η \frac{d L}{d θ}$

WHAT. We now know which way to step. But we also want steeper ground to give a bigger step (we're far from the bottom, hurry up) and gentle ground to give a small step (we're close, tiptoe). The slope's size already measures steepness! So use the slope itself for both jobs — its sign gives direction (Step 3), its size gives step length.

Multiply the slope by $- 1$ (to flip toward downhill) and by a small positive number $η$ (to keep the step sensible in size):

$Δ θ = - η \frac{d L}{d θ}$

Term by term, right where each symbol sits:

$Δ θ$ — the change in the knob (" $Δ$ ", delta, means "change in"). How far and which way we slide.
$-$ — the "go opposite the slope" flip from Step 3. Downhill, always.
$η$ — the learning rate (Greek "eta"), a small positive number like $0.1$ . A dial for how bold each step is.
$\frac{d L}{d θ}$ — the slope: carries direction (its sign) and steepness (its size) for free.

WHY multiply, rather than just take a fixed step left/right? Because a fixed step wastes information. The slope is a rich signal — steep places scream "big step" and flat places whisper "we're nearly there". Using $\frac{d L}{d θ}$ directly makes the algorithm slow down automatically as it nears the bottom.

PICTURE. Two points on the bowl. A steep point produces a long blue step arrow; a gently-sloped point near the bottom produces a short arrow. Same formula, self-scaling steps.

Step 5 — Assemble the update rule and take repeated steps

WHAT. "New knob position = old position + the change we computed." Using the step counter $t$ from the opening (so $θ_{t}$ is now, $θ_{t + 1}$ is next), substitute $Δ θ$ :

$θ_{t + 1} = θ_{t} + Δ θ = θ_{t} - η \frac{d L}{d θ}_{θ_{t}}$

Term by term:

$θ_{t}$ — where we stand now (the knob at step $t$ ; start at $t = 0$ ).
$θ_{t + 1}$ — where we stand after this step.
$- η \frac{d L}{d θ}_{θ_{t}}$ — the downhill nudge from Step 4, with the slope measured at our current spot $θ_{t}$ (that's what the $_{θ_{t}}$ means: "evaluated here").

WHY repeat? One step only uses the local tangent, which is honest only nearby. After moving, we're somewhere new with a new slope, so we re-feel the ground and step again. Repeat until the slope is (near) zero — flat ground, the valley floor.

PICTURE. A staircase of dots walking down the right wall of the bowl: $θ_{0} \to θ_{1} \to θ_{2} \to \dots$ , each step shorter than the last as the ground flattens, converging on $θ^{*}$ .

Let's trace it with numbers. Take the parent's Example 1: $L (θ) = θ^{2}$ . To get its slope we need one small differentiation rule, stated plainly so we don't smuggle it in:

So the slope is $2 θ$ . Start $θ_{0} = 4$ , $η = 0.1$ :

$θ_{1} = 4 - 0.1 \cdot 2 (4) = 4 - 0.8 = 3.2$ $θ_{2} = 3.2 - 0.1 \cdot 2 (3.2) = 3.2 - 0.64 = 2.56$ $θ_{3} = 2.56 - 0.1 \cdot 2 (2.56) = 2.56 - 0.512 = 2.048$

Each new position is $0.8$ times the old one: $θ_{t} = 4 \cdot (0.8)^{t}$ , sliding smoothly toward $θ^{*} = 0$ . The steps shrink on their own — exactly what Step 4 promised.

Step 6 — The edge case: what if $η$ is too big?

WHAT. The formula is honest only inside the little zone where the tangent line still resembles the curve. If $η$ makes the step jump out of that zone, we can leap clean over the valley and land higher on the far wall.

We first need one new tool — the "slope of the slope". The derivative of $L$ tells us the curve's steepness. If we ask "how fast is that steepness itself changing as we move?", we take the derivative again. That is the second derivative.

WHY it breaks — worked out for the bowl. Take $L = θ^{2}$ , whose slope is $2 θ$ . The update becomes

$θ_{t + 1} = θ_{t} - η (2 θ_{t}) = (1 - 2 η) θ_{t} .$

So each step multiplies the knob by the fixed number $(1 - 2 η)$ . Read term by term: the " $2$ " that appears is exactly the second derivative $\frac{d ^{2} L}{d θ ^{2}} = 2$ we just computed — the fixed rate at which the slope steepens.

The knob shrinks toward $0$ only if that multiplier is closer to zero than $1$ in size, i.e. $∣1 - 2 η ∣ < 1$ . Solving: $0 < η < 1$ . So the safe-step rule for this bowl is $η < \frac{2}{2} = 1$ — the " $2$ " on top is the fixed multiplier factor, the " $2$ " on the bottom is the second derivative.

Now watch it fail with $η = 1.1 > 1$ , starting $θ_{0} = 1$ :

$θ_{1} = 1 - 1.1 \cdot 2 (1) = 1 - 2.2 = - 1.2$ $θ_{2} = - 1.2 - 1.1 \cdot 2 (- 1.2) = - 1.2 + 2.64 = 1.44$

The knob bounces $1 \to - 1.2 \to 1.44 \to \dots$ , each bounce bigger — it diverges. The direction was right (downhill) but the distance overshot.

With $η = 0.1$ the multiplier is $1 - 0.2 = 0.8$ (size $< 1$ ) → converge, matching Step 5. With $η = 1.1$ it is $1 - 2.2 = - 1.2$ (size $> 1$ ) → blow up.

PICTURE. Same bowl, two walkers: a chalk-blue one with small $η$ spiralling gently into the base, a chalk-pink one with big $η$ zig-zagging outward up the walls.

Step 7 — The degenerate case: flat ground ( $slope = 0$ )

WHAT. When the slope is exactly $0$ , the update does nothing:

$θ_{t + 1} = θ_{t} - η \cdot 0 = θ_{t} .$

We stop. Good — if we're at the valley floor, staying put is correct. A point where the slope is zero is called a critical point.

WHY watch out. A zero slope means "flat", but flat happens at three kinds of critical point: the bottom of a valley (a minimum, great), the top of a hill (a maximum — we don't want to stop there), and a plateau / saddle where the ground is flat but the true minimum is elsewhere. Gradient descent freezes at any of them because it only feels the local slope; it cannot tell which kind it is from the slope alone.

How to tell them apart — the second-derivative test. We already met the second derivative $\frac{d ^{2} L}{d θ ^{2}}$ in Step 6: it measures curvature. At a critical point (where $\frac{d L}{d θ} = 0$ ) it gives us a test:

How to detect a stall in practice. You cannot see the whole curve, so you watch the numbers: if the loss $L$ stops decreasing while its slope is (near) zero, you have parked at a critical point. Then check the curvature sign as above — positive means you succeeded (a minimum); non-positive means you may be stuck at a maximum, plateau, or saddle point and should escape. The standard escapes are: add a nudge of momentum so you coast through flat spots (Momentum and Adam), inject randomness so you don't sit exactly on the flat (Stochastic Gradient Descent), or use curvature information directly (Second-Order Methods). In a simple convex bowl none of this is needed — the only critical point is the true minimum.

PICTURE. Three flat-tangent critical points on a wavy curve, each labelled with its curvature sign: a real minimum (pink, $\frac{d ^{2} L}{d θ ^{2}} > 0$ , good stop), a saddle/plateau (yellow, $\frac{d ^{2} L}{d θ ^{2}} = 0$ , false stop), and a maximum (blue, $\frac{d ^{2} L}{d θ ^{2}} < 0$ ). All share slope $= 0$ ; only the pink one is where we want to land.

The one-picture summary

Everything at once: the bowl $L (θ)$ ; at the current spot the tangent line gives the slope (Step 2); the slope's sign picks the downhill direction (Step 3); the slope's size times $- η$ sets the step length (Step 4); repeating draws the shrinking staircase to $θ^{*}$ (Step 5); too-big $η$ would fling you outward (Step 6); flat ground halts you, and the curvature sign tells minimum from maximum from plateau (Step 7).

Recall Feynman retelling — the whole walkthrough in plain words

You're standing somewhere on a curved bowl and you want the bottom, but you can only feel the ground right under your feet. So you feel the tilt — that's the slope. If it tilts up to your right, downhill is left; if it tilts up to your left, downhill is right. Either way, you step against the tilt — that's the minus sign. You make your step big when the ground is steep (you're far away, so hurry) and small when it's gentle (you're close, so tiptoe) — and the slope's own size gives you that for free. You multiply by a little caution dial called the learning rate so you don't leap too far. Then you feel the ground again from your new spot and repeat, taking smaller and smaller steps, until the ground goes flat and you stop. To be sure the flat spot is really the valley floor and not a hilltop or a plateau, you check the curve of the curve — the second derivative: cup-shaped (positive) means a true minimum, cap-shaped (negative) means a hilltop, and dead-flat (zero) means "look harder, could be a plateau or saddle". The only ways it goes wrong: crank the caution dial too high and you catapult past the valley and bounce out, or the ground goes flat somewhere that isn't the true bottom and you freeze there by mistake.

Recall

Why do we step in the direction opposite to the slope? ::: Positive slope means "right is uphill", so downhill is left; negative slope means the reverse. In both cases downhill is the opposite sign of the slope — the minus sign in the rule. What single quantity gives us both the step's direction and its size? ::: The slope (derivative) $\frac{d L}{d θ}$ : its sign is direction, its magnitude is steepness/size. What does the learning rate $η$ control, and why must it be positive? ::: How bold each step is. It must be positive; a negative $η$ would cancel the minus sign and send us uphill. What happens to the update when the slope is exactly zero? ::: $θ_{t + 1} = θ_{t}$ — no move; we've hit a critical point (flat ground) and halt. At a critical point, how do you tell a minimum from a maximum? ::: The second-derivative test: $\frac{d ^{2} L}{d θ ^{2}} > 0$ → minimum (cup up); $< 0$ → maximum (cap down); $= 0$ → inconclusive (plateau/saddle). For $L = θ^{2}$ with $θ_{0} = 4, η = 0.1$ , what is $θ_{1}$ ? ::: $4 - 0.1 \cdot 2 (4) = 4 - 0.8 = 3.2$ .