Gradients and directional derivatives: Visual walkthrough

Step 1 — The hill, the spot, and the question

WHAT. We have a landscape. Picture it first, then we name things.

PICTURE. Below, the wavy sheet is a hill. The black dot on the sheet is where we stand; the black dot on the flat floor beneath it is our ground position. Look at the two floor axes: one runs east, one runs north — a ground position is just "how far east, how far north."

Now that the picture is in front of you, we name what it shows. Give every ground position a pair of numbers $(x, y)$ — the east amount and the north amount you just saw on the axes. Above each such position the hill has a height, and we call that height $f (x, y)$ : the letter $f$ is a machine — feed it a floor position, it hands back the height of the sheet there. The floor dot in the figure is the fixed position we stand over; we name it $a = (a_{x}, a_{y})$ . The bold symbol $a$ (bold = "this is a pair of numbers, an arrow-able thing", not a single number) simply means "the spot we stand at."

WHY. Machine learning trains by rolling downhill on an error landscape, so the very first thing we need is a precise way to say "how high am I, and how does that height change as I move?" Everything else is answering that.

Step 2 — Pick a walking direction $u$

WHAT. From the spot $a$ we choose a way to walk. A direction on the floor is itself a pair of numbers $u = (u_{x}, u_{y})$ : " $u_{x}$ east and $u_{y}$ north for each unit you step." We insist the arrow has length exactly 1. Length of $u$ is written $∥ u ∥$ (the double bars mean "how long is this arrow"), and we compute it by Pythagoras: $∥ u ∥ = u_{x}^{2} + u_{y}^{2}$ . We demand $∥ u ∥ = 1$ .

WHY the length-1 rule. We want a slope per one metre walked. If our step-arrow were 2 metres long, every change would look twice as big — we'd be measuring "how far" tangled together with "how steep". Fixing the length to $1$ strips out "how far" and leaves pure steepness. This single choice is the reason a certain factor drops out later.

PICTURE. The red arrow is $u$ , drawn on the floor, exactly one unit long. The dashed circle shows all allowed directions — every red arrow we could pick has its tip on this circle.

Step 3 — Turn the 2-D walk into a 1-D walk: the slice $g (t)$

WHAT. As we walk, let $t$ be how many metres we have travelled ( $t = 0$ at the start). Our floor position after walking $t$ metres is $x (t) = a + t u,$ which in plain coordinates means $x (t) = a_{x} + t u_{x}$ and $y (t) = a_{y} + t u_{y}$ . Now define a brand-new one-input machine: $g (t) = f (a + t u) .$ So $g$ says: "give me metres walked $t$ , I give back the height there."

WHY. A hill is a hard 2-D object. But walking a straight line on it is a simple 1-D story: height versus distance, like a cross-section of the hill. We already know how to differentiate ordinary one-variable functions, so we shrink the hard problem to an easy one we can attack.

PICTURE. Left: the floor with the straight red walking line $a + t u$ . Right: the same walk seen as a curve — horizontal axis is $t$ (metres walked), vertical axis is $g (t)$ (height). The slope of this red curve at the start, $t = 0$ , is the number we want. That slope is written $g^{'} (0)$ — the prime $^{'}$ means "rate of change," and the $0$ means "measured right at the starting spot."

Step 4 — Why partial derivatives are the natural pieces

WHAT. A partial derivative asks a narrower question: "if I nudge only east and freeze north, how fast does height change?" We write it $\frac{\partial f}{\partial x}$ (the curly $\partial$ is a "d" that whispers other variables held constant). Likewise $\frac{\partial f}{\partial y}$ freezes east and nudges north.

WHY. Our walk mixes east-motion and north-motion together. To handle the mix we first measure each pure ingredient — the east-slope and the north-slope — then combine them. Partials are exactly those two pure ingredients.

PICTURE. Two thin cross-sections through our spot: one cutting purely east–west (its slope is $\partial f / \partial x$ ), one purely north–south (slope $\partial f / \partial y$ ). The red tangent segment on each shows the slope that partial measures.

Step 5 — The chain rule stitches the ingredients together

WHAT. We need $g^{'} (t)$ , and $g (t) = f (x (t), y (t))$ is "a function inside a function." The tool for differentiating a function-of-a-function is the Chain rule. It says: the total rate of change is the sum, over each coordinate, of (how fast $f$ changes with that coordinate) times (how fast that coordinate changes with $t$ ). Because our position moves as $t$ grows, each partial must be read at the current point $(x (t), y (t))$ — that is where we are right now, not back at the start: $g^{'} (t) = \frac{\partial f}{\partial x} (x (t), y (t)) \cdot \frac{d x}{d t} + \frac{\partial f}{\partial y} (x (t), y (t)) \cdot \frac{d y}{d t} .$

WHY this tool and not another. No single-variable rule handles two moving inputs at once. The chain rule is precisely the rule for "inner things moving, outer thing responding." It is the only tool that lets the east-slope and north-slope contribute simultaneously.

Now the two inner rates are trivial, because $x (t) = a_{x} + t u_{x}$ changes with $t$ at the constant rate $u_{x}$ , and $y (t) = a_{y} + t u_{y}$ at rate $u_{y}$ : $\frac{d x}{d t} = u_{x}, \frac{d y}{d t} = u_{y} .$ Substitute: $g^{'} (t) = \frac{\partial f}{\partial x} (x (t), y (t)) u_{x} + \frac{\partial f}{\partial y} (x (t), y (t)) u_{y} .$

Term by term: $\frac{\partial f}{\partial x}$ is the east-slope; $u_{x}$ is how much of your step is east; their product is the height gained from the eastward part of the step. The second term is the same story going north. Add them — total height gained per metre.

We only want the slope at the start, so set $t = 0$ . Then the current point is exactly our spot $a$ , i.e. $(x (0), y (0)) = (a_{x}, a_{y})$ , and the partials are evaluated at $a$ : $g^{'} (0) = \frac{\partial f}{\partial x} (a) u_{x} + \frac{\partial f}{\partial y} (a) u_{y} .$

PICTURE. The step arrow $u$ split into its east piece $u_{x}$ and north piece $u_{y}$ ; each piece is multiplied by its matching slope, and the two contributions stack up to the total.

Step 6 — Recognize the pattern: it's a dot product

WHAT. Collect the two slopes into one arrow, the gradient: $\nabla f (a) = (\frac{\partial f}{\partial x} (a), \frac{\partial f}{\partial y} (a)) .$ The upside-down triangle $\nabla$ ("nabla") just means "the vector of all the partials." Then the sum from Step 5 is exactly the dot product of two arrows — multiply matching components, add: $\nabla f (a) \cdot u = \frac{\partial f}{\partial x} (a) u_{x} + \frac{\partial f}{\partial y} (a) u_{y} .$

WHY name it a dot product. Because the dot product has a geometric meaning we can exploit in the next step. Rewriting the plain sum as $\nabla f \cdot u$ isn't new maths — it's re-reading the same number so its geometry becomes visible.

PICTURE. Two arrows on the floor — the gradient $\nabla f$ (red) and the step $u$ (black) — with the dot-product recipe "same-direction pieces multiply and add" annotated.

Step 7 — The geometry: $cos θ$ reveals steepest ascent

WHAT. Every dot product also equals "length times length times the cosine of the angle between the arrows": $\nabla f \cdot u = ∥\nabla f ∥ ∥ u ∥ cos θ .$ Here $θ$ is the angle between our step $u$ and the gradient, and $cos θ$ is a dial running from $1$ (arrows aligned) down to $- 1$ (arrows opposite). Since we forced $∥ u ∥ = 1$ , that factor vanishes: $D_{u} f = ∥\nabla f ∥ cos θ .$

WHY this form. It isolates the only thing we control — the angle $θ$ . The steepness we experience is just the fixed number $∥\nabla f ∥$ scaled by the dial $cos θ$ .

PICTURE. The gradient fixed; the red step arrow swept around a circle; a side gauge shows $cos θ$ and how the resulting slope grows to a max at $θ = 0$ .

Read the three landmark cases straight off the dial:

angle $θ$	$cos θ$	$D_{u} f$	meaning
$0$	$+ 1$	$+ ∥\nabla f ∥$	walk along $\nabla f$ → steepest ascent
$π$	$- 1$	$- ∥\nabla f ∥$	walk opposite $\nabla f$ → steepest descent (this is $- \nabla f$ , used by Gradient descent)
$π /2$	$0$	$0$	walk perpendicular → height unchanged → along a contour

Step 8 — The degenerate cases (never left guessing)

WHAT & WHY. A careful derivation must survive the awkward inputs. Three matter here.

Along a contour ( $θ = π /2$ ). Then $D_{u} f = 0$ : the ground stays level. This is why $\nabla f$ is perpendicular to the level curves / contours — the only directions giving zero slope are the ones at a right angle to the gradient.
A flat spot: $\nabla f = 0$ . If both partials are zero (a peak, a valley bottom, or a saddle), then $∥\nabla f ∥ = 0$ , so $D_{u} f = 0$ for every direction. There is no "uphill" — a summit, or the very minimum Gradient descent is chasing. The "steepest ascent direction" is undefined precisely because all directions tie at zero.
A non-unit step (the classic slip). If you forget and use an arrow with $∥ u ∥ = 2$ , the $∥ u ∥$ factor no longer disappears; $D_{u} f = ∥\nabla f ∥ \cdot 2 \cdot cos θ$ — a slope inflated by the arrow length. The number stops meaning "per metre." Always divide by the length first.

PICTURE. Left: circular contours with the red gradient stabbing across them at right angles. Middle: a flat summit where the gradient arrow shrinks to a dot. Right: a length-2 arrow with its inflated (wrong) slope crossed out beside the corrected unit version.

The one-picture summary

Everything compressed: the floor with our spot $a$ , the red gradient arrow, a fan of candidate step directions, and beside each the height it buys — biggest along the gradient, zero across it, most-negative opposite it. This single image is $D_{u} f = ∥\nabla f ∥ cos θ$ .

Recall Feynman retelling (plain words)

You're standing on a bumpy hill. You want to know how steeply the ground rises if you take a step in some chosen direction. First you measure two simple things: how steep it is if you step purely east, and how steep if you step purely north. Those two numbers, bundled into one arrow, are the gradient. Now any real step is part-east, part-north; the chain rule just says "take the east-steepness times your east-part, plus north-steepness times your north-part." That total is the gradient arrow "dotted" with your step arrow. And a dot product is secretly "how much your step lines up with the gradient." So: step exactly along the gradient → maximum climb; step sideways to it → you stay level (that's a contour ring); step straight against it → fastest way down, which is exactly the direction machine-learning training walks. Keep your step arrow one metre long so every number means "steepness per metre." That's the whole story.

Connections

Partial derivatives — the east/north ingredients of Step 4.
Chain rule — the stitching tool of Step 5.
Dot product and projections — the geometry unlocked in Steps 6–7.
Level sets and contours — the perpendicularity of Step 8.
Gradient descent — walks along $- \nabla f$ , the $θ = π$ case.
Jacobian and Hessian — where these partials generalize next.