Foundations — Gradients and directional derivatives

Before you can trust that arrow, you must be able to read every symbol on the page without hesitation. This note walks each one from nothing — plain words, a picture, and why the topic can't live without it. Read top to bottom; each piece is a brick for the next.

0. The function $f$ — the landscape itself

The picture: imagine a floor marked with an east axis $x$ and a north axis $y$. Above every floor spot $(x,y)$ the function raises the ground to height $f(x,y)$. The result is a surface — a hill. Figure 1 draws exactly this: the amber dot is a floor spot $(x,y)$ and the dashed amber line rises to its height on the surface.

Why the topic needs it. Everything else — slope, gradient, descent — is a question about this surface. No surface, nothing to be steep.

1. The symbols $x_1,\dots ,x_n$ and ${\mathbb{R}}^n$ — where you stand

Sometimes there are more than two inputs. Instead of naming them $x,y,z,\dots$ we number them $x_1,x_2,\dots ,x_n$. The little number below is a label, not a power — $x_2$ means "the second input", not $x$ squared.

Picture: ${\mathbb{R}}^2$ is an infinite sheet of graph paper (two coordinates). ${\mathbb{R}}^3$ is all of space. ${\mathbb{R}}^n$ is the same idea with $n$ coordinates — impossible to draw past 3, but the arithmetic is identical.

Why the topic needs it. ML models have thousands of inputs (weights). We must talk about "input number $i$" generally, so $x_i$ and ${\mathbb{R}}^n$ are unavoidable.

2. Vectors and $\mathbf{a}$, $\mathbf{u}$ — points and arrows

• $\mathbf{a}$ = the place you are standing on the floor.
• $\mathbf{u}$ = the direction you decide to step (an arrow).

Figure 2 shows both: the cyan arrow $\mathbf{a}$ marks where you stand, and the amber arrow $\mathbf{u}$ (of length exactly $1$) is a step you might take — the dotted sides $0.6$ and $0.8$ are its components.

Why unit vectors matter. When we ask "how fast does height change as I step in direction $\mathbf{u}$?", we want the answer per one metre walked. If the arrow were 2 metres long, we'd accidentally double the answer. Forcing length $1$ separates "which way" from "how far."

See Dot product and projections for how norms combine with angles next.

3. The limit ${\lim }_{h\to 0}$ — "zoom in until it's straight"

Slope means "rise over run." On a curve the rise-over-run depends on how big a step you take. To pin down the slope exactly at a point, we shrink the step $h$ toward $0$.

Picture: take two nearby points on a curve, draw the straight line through them (a secant), then slide the second point toward the first. The secant tips over until it just kisses the curve — that final line is the tangent, and its steepness is the derivative. In Figure 3 the faint white lines are secants for shrinking $h$; the bold amber line is the tangent they home in on.

Why the topic needs it. Both the ordinary derivative and the partial derivative are defined by this limit. It is the ground floor of all slopes.

4. The derivative and the partial symbol $\partial$

In one variable, the derivative $\frac{df}{dx}$ is the limit above: the instant slope of the curve $f(x)$.

With several inputs we need to say "slope along one axis only, holding the others frozen." That is the partial derivative, and it gets a special curly-d symbol $\partial$ (read "partial" or "del-dee") to warn you: other variables are being held constant.

Picture: stand on the hill and slice it with a vertical wall running exactly east–west. The wall cuts the surface into a single curve. $\frac{\partial f}{\partial x}$ is the slope of that curve. Turn the wall north–south and you get $\frac{\partial f}{\partial y}$.

Why the topic needs it. The gradient is built entirely out of these numbers — one per axis. Master this and see Partial derivatives for practice.

5. The gradient symbol $\nabla f$ — bundling all the slopes

Picture: at your feet, each axis-slope becomes one component of a single arrow. Stacked together, that arrow ends up pointing straight up the steepest direction of the hill, and its length is that steepest slope. (The why is proven in the parent note via the dot product; here we just make sure you can read $\nabla f$ as "the vector of partials.")

Why the topic needs it. It is the whole point. Gradient descent follows $-\nabla f$ downhill to train models.

6. Summation $\sum$ and the dot product $\cdot$

Picture: the dot product measures how much two arrows point the same way. Same direction → large positive. Perpendicular → zero. Opposite → large negative.

Why the topic needs it. The star quantity of this topic, the directional derivative (defined next), is a dot product. Both the algebra form ($\sum$) and the geometry form ($\cos \theta$) are used.

7. The directional derivative $D_{\mathbf{u}}f$ — the payoff symbol

Everything so far was built to write down one symbol you will meet constantly in the parent note.

Picture: stand at $\mathbf{a}$ (Figure 2's cyan arrow), face along $\mathbf{u}$ (the amber arrow), and read how fast the ground rises straight ahead. That reading is $D_{\mathbf{u}}f(\mathbf{a})$.

Why the topic needs it. It is the question the whole topic answers: "step this way — how fast do I rise?" Steepest ascent, steepest descent, and contour-walking are all special cases of this one number.

Prerequisite map

The diagram below is a prerequisite map: read it top-to-bottom. Each box is one idea from the sections above, and each arrow means "the idea at the tail is needed to build the idea at the head." Follow the arrows and you rebuild the whole topic; the words on an arrow say what the lower idea contributes.

Each box must feel obvious before you tackle the parent topic. If any box is fuzzy, reread its section above.

8. The Greek letter $\eta$ — step size

Picture: the gradient tells you which way to go; $\eta$ tells you how far per step. It is used in Gradient descent, and appears in the map above feeding the "gradient descent" box.

Equipment checklist