Foundations — Gradient descent intuition and update rule

Before you can read the parent note Gradient Descent, you need to own every piece of notation it throws at you. This page builds each one from nothing, in an order where each idea leans on the one before it. Nothing here is assumed.

1. The parameter $\theta$ — the "settings knob"

Picture a radio with dials. Each dial is one number you can turn. Turning the dials changes the sound. In machine learning, $\theta$ is all the dials at once.

• If there is one dial, $\theta$ is a single number, e.g. $\theta =4$.
• If there are many dials, $\theta$ is a list, e.g. $\theta =(m,b)$ for a line $y=mx+b$.

Why the topic needs it: gradient descent's whole job is deciding how to turn these dials. Without a name for "the thing we adjust," there is nothing to adjust.

Read the figure: the horizontal axis is the single dial $\theta$; the blue curve is a valley. Look at the yellow dot at $\theta =-3$ and the red dot at $\theta =1.5$ — those are two different dial settings, each sitting at a different height. The green star at the very bottom is ${\theta }^{\ast }$ (theta-star). The little star ${}^{\ast }$ always means "the best one" — the dial setting that makes us least wrong.

2. The loss $L(\theta )$ — the "wrongness meter"

The notation $L(\theta )$ reads "L of theta" — it means "feed the dial settings $\theta$ into the wrongness meter $L$, and out comes a number." This is function notation: the letter outside the brackets is a machine, the thing inside the brackets is what you feed it.

Read the figure below: every horizontal position is a choice of dials $\theta$. The height of the blue ground at that spot is the loss $L(\theta )$. Turning the dials = walking left or right; the height = how wrong you are.

• The red dot on the left sits high up → large $L$ → very wrong.
• The green star sits in the dip → small $L$ → nearly right.
• The lowest valley point → ${\theta }^{\ast }$ → the best settings.

Why the topic needs it: "learning" is the act of finding the valley of $L$. See Loss Functions for how these meters are actually built.

3. Slope — how steep the ground is

Imagine standing on a ramp. Walk 1 metre forward (run), and you rise 2 metres (rise). The slope is $\frac{\text{rise}}{\text{run}}=\frac{2}{1}=2$. Walk forward and drop 2 metres → slope $=-2$. Perfectly flat → slope $=0$.

Read the figure: it shows a run-of-1 triangle drawn on three spots of the same hill. On the left (yellow) the ground climbs — the triangle rises, slope $=+2$. In the middle (green) the ground is flat — no rise, slope $=0$. On the right (red) the ground falls — the triangle drops, slope $=-2$. Same picture, three signs.

The sign of the slope is the crucial part:

• slope $>0$: ground goes up as you move right → to go down, move left.
• slope $<0$: ground goes down as you move right → to go down, move right.
• slope $=0$: you're on flat ground — possibly a valley bottom.

Why the topic needs it: the entire strategy is "walk in the downhill direction." Slope is what tells you which way is downhill without seeing the whole mountain.

4. The derivative $\frac{dL}{d\theta }$ — slope, made exact

Why a new tool and not just slope? A ramp has one slope everywhere. But the loss landscape is curved — its steepness changes at every point. Ordinary "rise over run" needs two points. The derivative is the trick that gives you the slope using an infinitely small run, so it works at a single point on a curve.

Why does $\frac{d}{d\theta }({\theta }^2)=2\theta$? Let's build it, not assert it. Take two points a tiny gap $h$ apart, $\theta$ and $\theta +h$, and compute rise-over-run between them: $\frac{(\theta +h{)}^2-{\theta }^2}{h}=\frac{{\theta }^2+2\theta h+h^2-{\theta }^2}{h}=\frac{2\theta h+h^2}{h}=2\theta +h.$ Now shrink the gap: as $h$ gets infinitely small ($h\to 0$), the leftover $h$ vanishes and we are left with exactly $2\theta$. That shrinking is the whole idea of a derivative — the slope you'd feel with a step so small it's a single point. The figure shows the two points closing up onto one tangent line.

Read the figure: the red dot sits at $\theta =4$; the yellow line is the tangent whose slope is $2(4)=8$ — steep and climbing, so to go down you step left. The green star at $\theta =0$ shows a flat tangent: slope $=0$, the valley bottom.

5. Partial derivative $\frac{\partial L}{\partial m}$ — slope in one direction

Picture standing on a real 2-D hillside. Face east and ask "how steep is it if I walk east only?" — that's one partial. Face north and ask again — that's another partial. Each partial isolates one direction.

Why the topic needs it: real models have thousands of dials. You cannot describe the downhill direction with a single slope; you need one slope per dial. That is precisely what the parent note computes in Example 2 with $\frac{\partial L}{\partial m}$ and $\frac{\partial L}{\partial b}$.

6. The directional derivative — steepness in any chosen direction

On a 2-D hill you can face infinitely many compass directions, not just east and north. Each direction has its own steepness. A beautiful fact ties them all together: the directional derivative in direction $\vec{u}$ equals the dot product (defined next section) of the gradient with $\vec{u}$: $\text{steepness along }\vec{u}=\nabla L\cdot \vec{u}=|\nabla L|\cos \alpha ,$ where $\alpha$ is the angle between $\vec{u}$ and the gradient. Because $\cos \alpha$ is largest ($=1$) when $\alpha =0$, the steepness is maximised when you walk exactly along $\nabla L$. That is the reason — not a magic assertion — that the gradient points straight uphill. We'll unpack the dot product properly in section 8; keep this promise in mind.

7. The gradient $\nabla L$ — the full downhill compass

This arrow is not just a bookkeeping list — from the directional-derivative fact above, it points somewhere:

Read the figure: the blue rings are contour lines (like a map's height lines) of a bowl-shaped loss. From the white "you are here" dot, the red arrow $\nabla L$ points outward, away from the centre — that's uphill. The green arrow $-\nabla L$ points inward toward the yellow star at the bottom — that's the direction gradient descent actually steps.

"Walk in the $-\nabla L$ direction" is gradient descent in one sentence. Backpropagation is simply a fast recipe for computing $\nabla L$ inside neural networks.

8. The dot product $\nabla L^{\top }\Delta \theta$ — measuring "same direction?"

The notation $\nabla L^{\top }\Delta \theta$ (a superscript $\top$ means "transpose" — for our purposes just a way of writing a dot product) asks: does my step $\Delta \theta$ point the same way as uphill?

Read the figure: all three green step arrows have the same length; only the angle $\alpha$ to the red gradient changes. The value written under each is the dot product with a gradient of length $2$ and step of length $1$.

A concrete number: take $|\nabla L|=2$ and a step of length $|\Delta \theta |=1$.

• If step points uphill ($\alpha =0°$, $\cos \alpha =+1$): dot $=2\cdot 1\cdot 1=+2$. Loss rises most. Bad.
• If step points sideways ($\alpha =90°$, $\cos \alpha =0$): dot $=2\cdot 1\cdot 0=0$. Loss unchanged.
• If step points downhill ($\alpha =180°$, $\cos \alpha =-1$): dot $=2\cdot 1\cdot (-1)=-2$. Loss drops most. Best!

Why the topic needs it: this is the proof that "opposite to the gradient" is optimal. The most-negative dot product happens exactly when your step is antiparallel to $\nabla L$ — which is why the update rule has a minus sign.

9. The step $\Delta \theta$, learning rate $\eta$, and iteration $t$

Putting all the symbols together gives the parent note's headline formula, and now every piece is earned:

Prerequisite map

Equipment checklist

Test yourself — cover the right side and answer aloud.