Gradient descent intuition and update rule: Exercises

Before we start, let us pin down every symbol you will see, so nothing appears unearned.

Definition The vocabulary, once and for all

$L (θ)$ — the loss: a number that says "how wrong are we", low is good.
$θ$ — a parameter (or a whole list of them $θ = (θ_{1}, θ_{2}, \dots)$ ). This is the "position on the mountain".
$θ_{t}$ — the parameter at iteration $t$ ; $t = 0, 1, 2, \dots$ counts the steps we have taken.
$θ^{⋆}$ — the optimum: the parameter value that makes $L$ smallest (the valley floor).
$\nabla L (θ)$ — the gradient: the list of slopes, one per parameter. It points uphill (steepest increase).
$η$ (Greek "eta") — the learning rate: a small positive number that sets step size.
$m, b$ — in a straight-line model $\overset{y}{^} = m x + b$ , the slope $m$ and intercept $b$ are the two parameters we tune.
$x_{i}, y_{i}$ — the $i$ -th data point: input $x_{i}$ and true target $y_{i}$ . The index $i$ runs over the dataset ( $i = 1, 2, \dots$ ). The prediction is $\overset{y}{^}_{i} = m x_{i} + b$ .
$A$ — a matrix of numbers; $I$ — the identity matrix (the matrix that leaves any vector unchanged, like multiplying by 1).
$λ_{m a x} (A)$ — the largest eigenvalue of $A$ : the biggest "stretch factor" $A$ applies along one of its natural axes.
The update rule: $θ_{t + 1} = θ_{t} - η \nabla L (θ_{t})$ . The minus sign turns "uphill" into "downhill".

Level 1 — Recognition

Recall Solution 1.1

Step size is controlled by $η$ (the learning rate) — it is just a scaling number.
Direction is controlled by $- \nabla L (θ_{t})$ — the gradient points uphill, the minus flips it downhill.
What it looks like: $\nabla L$ is the arrow, $η$ is how far along that arrow you slide.

Recall Solution 1.2

$\nabla L (θ) = \frac{d}{d θ} θ^{2} = 2 θ$ (power rule: bring the 2 down, drop the exponent by 1).
At $θ = 3$ : $\nabla L (3) = 2 \cdot 3 = 6$ .
Meaning: at $θ = 3$ the hill rises $6$ units of loss per unit of $θ$ — steep and pointing right, so we step left.

Recall Solution 1.3

False. The gradient points toward steepest increase (uphill, away from the minimum). We deliberately walk in the opposite direction $- \nabla L$ to go downhill. That is exactly why the update rule has a minus sign.

Level 2 — Application

Recall Solution 2.1

Gradient: $\nabla L (θ) = 2 θ$ .

$θ_{1} = 5 - 0.1 (2 \cdot 5) = 5 - 1 = 4$
$θ_{2} = 4 - 0.1 (2 \cdot 4) = 4 - 0.8 = 3.2$
$θ_{3} = 3.2 - 0.1 (2 \cdot 3.2) = 3.2 - 0.64 = 2.56$
Pattern: each step multiplies by $(1 - 0.1 \cdot 2) = 0.8$ , so $θ_{t} = 5 \cdot (0.8)^{t}$ , sliding toward $θ^{⋆} = 0$ .

Recall Solution 2.2

$θ_{1} = 5 - 1.0 (2 \cdot 5) = 5 - 10 = - 5$
$θ_{2} = - 5 - 1.0 (2 \cdot (- 5)) = - 5 + 10 = + 5$
Observation: it bounces $5 \to - 5 \to 5 \to \dots$ forever, never shrinking. The multiplier is $(1 - 1.0 \cdot 2) = - 1$ , magnitude $1$ , so no progress — a perfect oscillation.

The figure below plots the bowl $L = θ^{2}$ (violet) and draws each update as a magenta arrow. Notice the arrows jump from the right wall to the left wall at the same height every time — the ball never descends because a multiplier of $- 1$ only mirrors the point across the minimum without shrinking it. This is the visual signature of " $η$ exactly at the stability edge".

Figure — Gradient descent intuition and update rule

Recall Solution 2.3

Shrinking requires the multiplier's size below 1: $∣1 - 2 η ∣ < 1$ .
Solve: $- 1 < 1 - 2 η < 1 \Rightarrow 0 < 2 η < 2 \Rightarrow 0 < η < 1$ .
So any $η$ in $(0, 1)$ converges; $η = 1$ oscillates (Ex 2.2); $η > 1$ diverges.

Level 3 — Analysis

Recall Solution 3.1

Predictions at start: $\overset{y}{^}_{1} = m x_{1} + b = 0, \overset{y}{^}_{2} = 0$ . Errors $e_{i} = \overset{y}{^}_{i} - y_{i}$ : $e_{1} = - 3, e_{2} = - 5$ .

$\frac{\partial L}{\partial m} = \sum_{i} e_{i} x_{i} = (- 3) (1) + (- 5) (2) = - 13$
$\frac{\partial L}{\partial b} = \sum_{i} e_{i} = - 3 + (- 5) = - 8$
Updates: $m_{1} = 0 - 0.01 (- 13) = 0.13$ , $b_{1} = 0 - 0.01 (- 8) = 0.08$ .
Why both rise? Gradients are negative, minus flips them positive — the line was flat at $0$ , both $m$ and $b$ climb to reduce error.

Recall Solution 3.2

First, where does the change formula come from? The first-order Taylor (linear) approximation says that for a small step $Δ θ$ , $L (θ + Δ θ) \approx L (θ) + \nabla L^{⊤} Δ θ,$ so the loss change is $Δ L \approx \nabla L^{⊤} Δ θ$ . This is the directional derivative — exact in the limit of infinitesimally small steps, which is why the argument below is a local one.

Rewrite the dot product geometrically: $\nabla L^{⊤} Δ θ = ∥\nabla L ∥ ∥Δ θ ∥ cos α$ , where $α$ is the angle between the gradient and the step.
We want this most negative. $∥\nabla L ∥, ∥Δ θ ∥$ are fixed positives, so only $cos α$ can be steered.
$cos α$ is smallest ( $= - 1$ ) at $α = 18 0^{\circ}$ — pointing exactly opposite the gradient. Hence $- \nabla L$ wins.

In the figure below, the violet arrow is the (uphill) gradient and the magenta arrow is $- \nabla L$ at $18 0^{\circ}$ from it; the orange arrow is some other step at angle $α$ . Because $Δ L$ scales with $cos α$ , only the magenta arrow reaches the most-negative value — every other direction leaks some of its length sideways where it does no downhill work. That "wasted sideways component" is exactly what the dot product measures.

Recall Solution 3.3

Wrong sign — they wrote $θ + η \nabla L$ (climbing uphill). Test: even a tiny $η$ still increases loss → sign bug.
$η$ too large — overshooting past the valley. Test: shrinking $η$ (e.g. $\times 0.1$ ) makes loss start decreasing → learning-rate issue, not a sign bug.

Level 4 — Synthesis

Recall Solution 4.1

$A$ is diagonal, so each coordinate evolves independently with multipliers $(1 - η \cdot 1)$ and $(1 - η \cdot 10)$ .

Coordinate 1 needs $∣1 - η ∣ < 1 \Rightarrow 0 < η < 2$ .
Coordinate 2 needs $∣1 - 10 η ∣ < 1 \Rightarrow 0 < η < 0.2$ .
Both converge only for the tighter bound: $0 < η < 0.2 = \frac{2}{λ _{m a x}}$ with $λ_{m a x} = 10$ . This is exactly the parent note's stability rule $η < 2/ λ_{m a x} (A)$ .

Recall Solution 4.2

Coordinate 1 multiplier: $1 - 0.15 \cdot 1 = 0.85$ (shrinks $15%$ per step).
Coordinate 2 multiplier: $1 - 0.15 \cdot 10 = - 0.5$ (shrinks to half, but flips sign each step).
Faster is coordinate 2: $∣ - 0.5 ∣ = 0.5 < 0.85$ . The steep (large-eigenvalue) direction is squeezed fastest — but it is also the one that would diverge first if $η$ grew. This tension is why ill-conditioned problems are hard, motivating Momentum and Adam.

Recall Solution 4.3

(a) → Stochastic Gradient Descent (mini-batch gradients).
(b) → Learning Rate Schedules (decay / warm-up).
(c) → Backpropagation (efficient chain-rule gradient computation).

Level 5 — Mastery

Recall Solution 5.1

Set-up. By the first-order Taylor approximation, the loss change for a small step is $Δ L \approx \nabla L^{⊤} Δ θ$ . We restrict all candidate steps to a fixed length $∥Δ θ ∥ = ϵ$ — geometrically this is a sphere of radius $ϵ$ around $θ$ : we are asking "of all directions I could step the same distance, which drops the loss most?" The constraint forces $Δ θ$ onto that sphere so that the comparison is fair (a longer step would trivially change $L$ more just by being longer).

Geometric argument (no calculus needed). Write $\nabla L^{⊤} Δ θ = ∥\nabla L ∥ ϵ cos α$ . On the sphere, $∥\nabla L ∥$ and $ϵ$ are constants; the only free quantity is $cos α \in [- 1, 1]$ . It is minimized at $cos α = - 1$ , i.e. $Δ θ$ antiparallel to $\nabla L$ .
Same result via Lagrange multipliers (for the sceptic). Minimizing $\nabla L^{⊤} Δ θ$ subject to $∥Δ θ ∥^{2} = ϵ^{2}$ gives the stationarity condition $\nabla L = 2 μ Δ θ$ for a multiplier $μ$ — so $Δ θ$ must be parallel to $\nabla L$ ; picking the sign that minimizes (rather than maximizes) selects the antiparallel one.
The antiparallel vector of length $ϵ$ is $Δ θ^{⋆} = - ϵ \frac{\nabla L}{∥\nabla L ∥}$ . Absorbing $ϵ /∥\nabla L ∥$ into $η$ recovers $Δ θ = - η \nabla L$ . This is why GD is called steepest descent. (See Convex Optimization for when this local step reaches the global minimum.)

Recall Solution 5.2

Gradient: $\nabla L = a θ$ . Update: $θ_{1} = θ_{0} - η a θ_{0} = (1 - η a) θ_{0}$ .
To hit $0$ in one step for any $θ_{0}$ : $1 - η a = 0 \Rightarrow η = \frac{1}{a}$ .
Check with $a = 2$ ( $L = θ^{2}$ ): $η = 0.5$ — matches the "fastest" note from the L2 trap. This $1/ a$ is precisely the Newton step, the bridge to Second-Order Methods.

Recall Solution 5.3

Gradient: $\nabla L = (2 x, - 2 y)$ .

From $(0.1, 0)$ : $\nabla L = (0.2, 0)$ . The $y$ -component is zero, so $y$ never moves — GD slides down the $x$ -bowl to $(0, 0)$ and stalls at the saddle, never noticing $y$ falls away.
From $(0.1, 0.1)$ : $\nabla L = (0.2, - 0.2)$ ; the $y$ -update is $y_{1} = 0.1 - 0.1 (- 0.2) = 0.12$ , so $y$ grows, escaping downhill.

The figure below draws the saddle's contour lines with two GD paths. The magenta path starts on the $x$ -axis and marches straight into the centre where it freezes — because the $y$ -slope is exactly zero there is no arrow pushing it off. The violet path starts a hair off-axis and its arrows curve outward along $y$ , rolling off the saddle. The geometry shows why symmetric starting points are dangerous: they hide the very direction that would let you escape.

Recall One-line summary to recite

Reveal each answer to test yourself. What is the update rule? ::: $θ_{t + 1} = θ_{t} - η \nabla L (θ_{t})$ — flip the uphill arrow, scale by $η$ , repeat. Stability bound for $\frac{1}{2} θ^{⊤} A θ$ ? ::: $η < 2/ λ_{m a x} (A)$ . Why is $- \nabla L$ steepest descent? ::: minimize $∥\nabla L ∥ ϵ cos α$ ; best at $cos α = - 1$ .