Gradient descent intuition and update rule: Worked examples

This page is a drill. The parent note told you the rule:

θ_{t + 1} = θ_{t} - η \nabla L (θ_{t})

Here $θ$ (theta) is the thing we are tuning (a number, or a list of numbers), $L$ is the loss (the "height on the hill"), $\nabla L$ (read "nabla L", the gradient) is the arrow pointing uphill, and $η$ (read "eta", the learning rate) is how big a step we take. One more symbol we lean on below:

If any of those words are shaky, read the parent first. Below we build a matrix of every case this one formula can throw at you, then we work an example for each cell so you never meet a scenario cold.

The scenario matrix

Every gradient-descent problem lives in one of these boxes. The columns are what makes it tricky; each row is a concrete example that lands in that box.

#	Case class	What is different here	Example
A	Start on the right of the minimum ( $θ_{0} > θ^{}$ ), gradient positive*	Step must go left (minus a positive)	Ex 1
B	Start on the left ( $θ_{0} < θ^{}$ ), gradient negative*	Step must go right (minus a negative)	Ex 2
C	Zero gradient — you start AT a minimum, or at a flat spot	Nothing moves; is it done or stuck?	Ex 3
D	Learning rate too big — overshoot / diverge	Loss grows; sign flips each step	Ex 4
E	2-D parameters — gradient is a vector	Update every coordinate at once	Ex 5
F	Real-world word problem — units and meaning	Translate story → $L$ , $\nabla L$ , step	Ex 6
G	Non-convex — many valleys	Which valley you land in depends on start	Ex 7
H	Exam twist — pick the largest safe $η$	Use the stability bound $η < 2/ λ_{m a x}$	Ex 8

Figure — Gradient descent intuition and update rule

Figure s01 — one downhill step on $L (θ) = θ^{2}$ : the step (white) points exactly opposite the gradient (amber).

Case A — start right of the minimum (positive gradient)

Worked example Ex 1 —

$L (θ) = θ^{2}$ , start $θ_{0} = 4$ , $η = 0.1$ Forecast: guess before reading — will $θ$ go up or down, and roughly where does it end?

Compute the gradient. $L (θ) = θ^{2}$ , so $\nabla L (θ) = 2 θ$ . Why this step? The update needs a slope; the power rule $\frac{d}{d θ} θ^{2} = 2 θ$ gives it. At $θ = 4$ this is $2 (4) = 8$ — a positive slope means the curve rises as we move right, so downhill is left. This is exactly the situation drawn in figure s01 above.
First update. $θ_{1} = 4 - 0.1 \times 8 = 4 - 0.8 = 3.2$ . Why this step? Minus a positive number moves us left, toward $0$ — exactly the direction of decreasing height.
Repeat. $\nabla L (3.2) = 6.4 \Rightarrow θ_{2} = 3.2 - 0.64 = 2.56$ . Then $θ_{3} = 2.56 - 0.512 = 2.048$ . Why this step? Notice the steps shrink as the slope shrinks — the ball naturally slows near the bottom.
See the pattern. Each step multiplies by $0.8$ : $θ_{t} = 4 (0.8)^{t}$ . Why this step? $θ_{t + 1} = θ_{t} - 0.1 (2 θ_{t}) = (1 - 0.2) θ_{t} = 0.8 θ_{t}$ . This is a geometric shrink toward $θ^{*} = 0$ .

Verify: $θ_{3} = 4 (0.8)^{3} = 4 (0.512) = 2.048$ ✓. Loss dropped from $16$ to $2.04 8^{2} \approx 4.19$ — smaller, good.

Case B — start left of the minimum (negative gradient)

Worked example Ex 2 — same

$L (θ) = θ^{2}$ , but start $θ_{0} = - 4$ , $η = 0.1$ Forecast: the minus sign is still there — will the step still push us toward zero from the left?

Gradient at the start. $\nabla L (- 4) = 2 (- 4) = - 8$ . Why this step? On the left branch the curve falls as we move right, so the slope is negative. Downhill is now to the right.
Update. $θ_{1} = - 4 - 0.1 (- 8) = - 4 + 0.8 = - 3.2$ . Why this step? Minus a negative becomes plus — the formula automatically flips direction. No special case needed; the sign of $\nabla L$ does the steering.
Confirm the mirror. $θ_{2} = - 2.56, θ_{3} = - 2.048$ , i.e. $θ_{t} = - 4 (0.8)^{t}$ . Why this step? This is the exact mirror image of Ex 1 — proving the rule works on both sides without us thinking about which side we're on.

Verify: $θ_{1} = - 3.2$ and $∣ θ_{1} ∣ < ∣ θ_{0} ∣$ ✓ (closer to $θ^{*} = 0$ ). The single formula handled positive and negative gradients identically.

Case C — the gradient is zero (start at the minimum / a flat spot)

Worked example Ex 3 — start

$θ_{0} = 0$ on $L (θ) = θ^{2}$ , and start $θ_{0} = 0$ on $L (θ) = θ^{3}$ Forecast: if the slope is $0$ , what does the update do? Is "nothing moves" always good news?

Bowl case, $L = θ^{2}$ . $\nabla L (0) = 2 (0) = 0$ , so $θ_{1} = 0 - η (0) = 0$ . Why this step? Zero gradient ⇒ zero step. We are exactly at the bottom $θ^{*} = 0$ , so staying is correct — this is convergence.
Flat-inflection case, $L = θ^{3}$ . $\nabla L (θ) = 3 θ^{2}$ , so $\nabla L (0) = 0$ again — the update also freezes at $0$ . Why this step? But $θ^{3}$ has no minimum at $0$ (it keeps falling for $θ < 0$ ). A zero gradient means "flat here", not "lowest here."
Diagnose the difference. Nudge left to $θ = - 0.1$ : for $θ^{2}$ the loss rises ( $0.01 > 0$ ) — true minimum. For $θ^{3}$ the loss drops ( $- 0.001 < 0$ ) — a flat inflection point (in 1-D), not a minimum. Why this step? Gradient descent gets stuck at any zero-gradient point. In one dimension we call such a non-minimum flat spot an inflection point; the multivariable cousin (flat in some directions, downhill in others) is a saddle point — see Local Minima and Saddle Points. Whether a zero-gradient point is actually good is a separate question from whether descent halts there.

Verify: $\nabla L (0) = 0$ for both, so both freeze ✓. Yet $L (- 0.1) = 0.01 > 0$ for $θ^{2}$ but $L (- 0.1) = - 0.001 < 0$ for $θ^{3}$ ✓ — zero gradient ≠ minimum.

Case D — learning rate too large (overshoot & divergence)

Worked example Ex 4 —

$L (θ) = θ^{2}$ , start $θ_{0} = 4$ , but now $η = 1.5$ Forecast: we found $η = 0.1$ shrinks by $0.8$ each step. What does $η = 1.5$ do — faster, or disaster?

One step. $\nabla L (4) = 8$ , so $θ_{1} = 4 - 1.5 (8) = 4 - 12 = - 8$ . Why this step? We wanted to nudge toward $0$ but we flew past it to the other side, and further out than we started ( $∣ - 8 ∣ > ∣4∣$ ).
Next step. $\nabla L (- 8) = - 16$ , so $θ_{2} = - 8 - 1.5 (- 16) = - 8 + 24 = 16$ . Why this step? Now we overshoot back the other way, even further. The loss $θ^{2}$ went $16 \to 64 \to 256$ — growing.
General multiplier. $θ_{t + 1} = (1 - η \cdot 2) θ_{t} = (1 - 3) θ_{t} = - 2 θ_{t}$ . Why this step? The step multiplier is $∣ - 2 ∣ = 2 > 1$ , so magnitude doubles each step and the sign flips — the signature of divergence. This runaway zig-zag is drawn in figure s02 below.
Stability threshold. Convergence needs $∣1 - 2 η ∣ < 1$ , i.e. $0 < η < 1$ . Any $η \geq 1$ fails. Why this step? This is the 1-D version of the parent's bound $η < 2/ λ_{m a x}$ ; here the relevant number is $2$ , so $η < 1$ . (What $λ_{m a x}$ means is unpacked in Ex 8.)

Verify: $θ_{1} = - 8, θ_{2} = 16$ ✓, and $∣1 - 2 (1.5) ∣ = 2 > 1$ ✓ (diverges). Contrast Ex 1's $∣1 - 2 (0.1) ∣ = 0.8 < 1$ (converges).

Figure s02 — $η = 1.5$ overshoots the valley: each iterate ( $θ_{0} \to θ_{1} \to θ_{2} \to θ_{3}$ ) lands higher and further out, so the loss explodes.

Case E — two parameters at once (the gradient is a vector)

Worked example Ex 5 — linear regression

$y = m x + b$ , data $(1, 3), (2, 5)$ , start $m_{0} = b_{0} = 0$ , $η = 0.01$ Forecast: now $θ = (m, b)$ is a pair. Which way does each coordinate move on the first step?

Loss: $L (m, b) = \frac{1}{2} \sum_{i} (m x_{i} + b - y_{i})^{2}$ , where $\overset{y}{^}_{i} = m x_{i} + b$ is our prediction.

Derive $\partial L / \partial m$ . Treat $b$ as frozen and differentiate one term $\frac{1}{2} (m x_{i} + b - y_{i})^{2}$ with respect to $m$ . The chain rule says: differentiate the outside ("half of something squared" → "something"), then multiply by the derivative of the inside $(m x_{i} + b - y_{i})$ with respect to $m$ , which is $x_{i}$ . Term-by-term: $\frac{\partial L}{\partial m} = \sum_{i} (m x_{i} + b - y_{i}) \cdot x_{i} .$ Why this step? We need to know how the loss reacts to wiggling $m$ alone; that is exactly what the partial derivative measures. The inner derivative is $x_{i}$ because only the $m x_{i}$ piece depends on $m$ .
Derive $\partial L / \partial b$ . Same trick, differentiating the inside $(m x_{i} + b - y_{i})$ with respect to $b$ , whose derivative is $1$ : $\frac{\partial L}{\partial b} = \sum_{i} (m x_{i} + b - y_{i}) \cdot 1 = \sum_{i} (m x_{i} + b - y_{i}) .$ Why this step? $b$ shifts every prediction up or down equally, so its inner derivative is $1$ — no $x_{i}$ factor. This is why the two partials differ by an $x_{i}$ weight.
Predictions at start. $\overset{y}{^}_{1} = 0 \cdot 1 + 0 = 0$ , $\overset{y}{^}_{2} = 0$ . Why this step? The gradient depends on the error $\overset{y}{^}_{i} - y_{i}$ ; we need the current predictions first.
Errors. $e_{1} = 0 - 3 = - 3$ , $e_{2} = 0 - 5 = - 5$ . Why this step? Negative errors mean we predict too low — the line should tilt up and lift.
The two partials, evaluated. $\frac{\partial L}{\partial m} = (- 3) (1) + (- 5) (2) = - 13$ , $\frac{\partial L}{\partial b} = - 3 + (- 5) = - 8$ . Why this step? Both are negative, so both parameters should increase to cut the error.
Update both together. $m_{1} = 0 - 0.01 (- 13) = 0.13$ , $b_{1} = 0 - 0.01 (- 8) = 0.08$ . Why this step? Each coordinate uses its own partial. A classic bug is to compute the two partials but then accidentally apply the same one to both parameters (e.g. subtracting $\partial L / \partial m$ from both $m$ and $b$ ) — always pair each parameter with its matching partial.

Verify: $m_{1} = 0.13, b_{1} = 0.08$ ✓. Sanity: new prediction $\overset{y}{^}_{2} = 0.13 (2) + 0.08 = 0.34 > 0$ — closer to the target $5$ than $0$ was ✓. See Backpropagation for computing such gradients at scale.

Case F — real-world word problem (units matter)

Worked example Ex 6 — a bakery's price setting

A bakery sells $θ$ loaves/day at price shaped so daily loss of profit is $L (θ) = (θ - 60)^{2}$ dollars², where $θ^{*} = 60$ is the ideal output. Today they make $θ_{0} = 90$ loaves. Use $η = 0.2$ loaves per (dollar²/loaf).

Forecast: should tomorrow's output rise or fall, and by how much?

Gradient with units. $\nabla L (θ) = 2 (θ - 60)$ , units $dollar^{2} / loaf$ . At $θ_{0} = 90$ : $2 (30) = 60$ . Why this step? Positive gradient ⇒ making more loaves raises profit-loss ⇒ we should make fewer.
Update. $θ_{1} = 90 - 0.2 \times 60 = 90 - 12 = 78$ loaves. Why this step? Units cancel: $loaf / (\frac{dollar ^{2}}{loaf}) \times \frac{dollar ^{2}}{loaf} = loaf$ — the answer is genuinely in loaves. Good.
Iterate the meaning. $θ_{2} = 78 - 0.2 \cdot 2 (18) = 78 - 7.2 = 70.8$ , heading to $θ^{*} = 60$ . Why this step? The multiplier is $1 - 0.2 \cdot 2 = 0.6$ , so each day the gap from $60$ shrinks by $40%$ — a business-friendly gradual glide.

Verify: $θ_{1} = 78$ ✓, and $∣78 - 60∣ = 18 < ∣90 - 60∣ = 30$ ✓ (closer to optimum). Units are loaves ✓.

Case G — non-convex: which valley you land in depends on the start

Worked example Ex 7 —

$L (θ) = θ^{4} - 4 θ^{2}$ (a double-well), $η = 0.05$ Forecast: this curve has two bottoms. Two different starts — do they reach the same place?

Gradient: $\nabla L (θ) = 4 θ^{3} - 8 θ = 4 θ (θ^{2} - 2)$ . Zeros at $θ = 0, \pm 2 \approx \pm 1.414$ . The two minima are at $θ^{*} = \pm 2$ ; $θ = 0$ is a local max.

Start A: $θ_{0} = 1$ . $\nabla L (1) = 4 (1) - 8 (1) = - 4$ ; $θ_{1} = 1 - 0.05 (- 4) = 1.2$ . Why this step? Negative slope pushes us right, toward the positive well $+ 2$ .
Start B: $θ_{0} = - 1$ . $\nabla L (- 1) = - 4 + 8 = + 4$ ; $θ_{1} = - 1 - 0.05 (4) = - 1.2$ . Why this step? Now the slope is positive, pushing us left, toward the negative well $- 2$ .
Conclusion. Same algorithm, same $η$ , but the basin of attraction decides the outcome — one start converges to $+ 2$ , the other to $- 2$ (figure s03 traces both paths). Why this step? In non-convex landscapes gradient descent finds a minimum, not the global one. Compare Convex Optimization (one bowl, no such ambiguity).

Verify: $\nabla L (2) = 4 (22) - 82 = 82 - 82 = 0$ ✓ (a genuine rest point). Step-1 signs: $1 \to 1.2$ (rightward) and $- 1 \to - 1.2$ (leftward) ✓.

Figure s03 — the double-well $L = θ^{4} - 4 θ^{2}$ : the amber path from $θ_{0} = 1$ falls into the right well $+ 2$ ; the white path from $θ_{0} = - 1$ falls into the left well $- 2$ . The cyan dot at $θ = 0$ is the local max separating the two basins.

Case H — exam twist: largest safe learning rate

Before the example, we owe you one definition — the symbol $λ_{m a x}$ (read "lambda-max") in the stability bound $η < 2/ λ_{m a x}$ .

Definition Eigenvalue and

$λ_{m a x}$ — "curvature along a direction" For a bowl-shaped loss $L (θ) = \frac{1}{2} θ^{⊤} A θ$ , the matrix $A$ describes how steeply the bowl curves in each direction. An eigenvalue $λ$ of $A$ is simply the curvature number along one special axis of that bowl: a big $λ$ means a steep, narrow direction; a small $λ$ means a shallow, wide direction. $λ_{m a x}$ (read "lambda-max") is the largest of these — the steepest, most sensitive direction. Why it sets the speed limit: along a direction with curvature $λ$ , one gradient step multiplies your distance-to-bottom by $(1 - η λ)$ . That shrinks the error only when $∣1 - η λ ∣ < 1$ , i.e. $η < 2/ λ$ . The steepest direction (largest $λ$ ) has the smallest such limit, so it is the one that decides whether the whole run is stable. For a plain 1-D $L = θ^{2}$ the single curvature number is $2$ — exactly the " $2$ " you saw in Ex 4's bound $η < 1$ .

Worked example Ex 8 — pick the biggest stable

$η$ for a stretched bowl $L (θ) = \frac{1}{2} θ^{⊤} A θ$ with $A = [10002]$ . What is the largest $η$ for which gradient descent still converges?

Forecast: two directions with different steepness — which one limits us?

Read the curvatures (eigenvalues). $A$ is diagonal, so its eigenvalues are just its diagonal entries: $λ = 10$ (steep axis) and $λ = 2$ (shallow axis). Why this step? For a diagonal $A$ , each coordinate curves independently, and its curvature is its own diagonal number — no extra computation needed. The stability bound cares about $λ_{m a x}$ , the steepest of these.
Apply the bound. $η < \frac{2}{λ _{m a x}} = \frac{2}{10} = 0.2$ . Why this step? Along the steep axis the update multiplier is $1 - η \cdot 10$ ; convergence needs $∣1 - 10 η ∣ < 1$ , i.e. $η < 0.2$ . Any bigger and the steep direction diverges even though the shallow one is fine.
The boundary $η = 0.2$ exactly. At $η = 0.2$ the steep-axis multiplier is $1 - 10 (0.2) = - 1$ , whose magnitude is exactly $1$ . Why this step? A multiplier of $- 1$ neither shrinks nor grows the coordinate — it just flips its sign each step, so the steep component oscillates forever at constant size instead of settling. That is why the bound is a strict inequality $η < 0.2$ : the endpoint is not convergence but perpetual bouncing.
Check the shallow axis is safe. At $η = 0.2$ : $∣1 - 2 (0.2) ∣ = 0.6 < 1$ ✓ — the $λ = 2$ direction converges comfortably; it never sets the limit. Why this step? Confirms the max eigenvalue, not the min, is the binding constraint — a classic exam trap.

Verify: $η_{m a x} = 2/10 = 0.2$ ✓. At $η = 0.2$ : steep multiplier $∣1 - 10 (0.2) ∣ = 1$ (borderline, oscillates), shallow $∣1 - 2 (0.2) ∣ = 0.6 < 1$ ✓. Use Learning Rate Schedules or Momentum and Adam to cope when $λ_{m a x} ≫ λ_{m i n}$ .

Recall Quick self-test

Which case has the step multiplier exactly $0.8$ ? ::: Ex 1/Ex 2 with $η = 0.1$ , $L = θ^{2}$ : $1 - 2 (0.1) = 0.8$ . Why does $θ_{0} = 0$ freeze on both $θ^{2}$ and $θ^{3}$ ? ::: Both have $\nabla L (0) = 0$ ; zero gradient ⇒ zero step, regardless of whether it's a true minimum. What is $λ_{m a x}$ in plain words? ::: The curvature of the loss along its steepest direction — the biggest eigenvalue of the curvature matrix; it sets the learning-rate limit $η < 2/ λ_{m a x}$ . In Ex 8, what happens exactly at $η = 2/ λ_{m a x} = 0.2$ ? ::: The steep-axis multiplier is $- 1$ , so that component oscillates at constant size forever — it does not converge; hence the bound is strict. In Ex 7, what decides which well you reach? ::: The sign of the starting point — its basin of attraction — since the landscape is non-convex.