Local vs global minima - maxima: Visual walkthrough

This page rebuilds the whole idea of local vs global extrema from a blank picture. We start with a single wiggly curve, ask "where are its bottoms and tops?", and slowly earn every symbol — slope, flat spots, curving up vs down — until we can classify any point. Then we climb into 2D and see why machine learning is hard.

A smart 12-year-old who has never seen a derivative should be able to follow from line one. Every symbol is defined the moment before we use it.

Step 1 — What is a "curve" and what are its bottoms and tops?

WHAT. Picture a landscape drawn as a single line. The horizontal position is a number we control — call it $x$ . The height of the line above that position is another number — call it $f (x)$ , read " $f$ of $x$ ", meaning "the height the machine $f$ gives back when you feed it $x$ ".

WHY. Before we can hunt for the lowest valley or highest peak, we must agree on what we are looking at: a rule that turns each horizontal spot into a height.

PICTURE. Look at the figure. The horizontal axis is $x$ (where you stand). The vertical axis is $f (x)$ (how high you are). The teal dot sits in a valley — lower than its neighbours. The orange dot sits on a peak — higher than its neighbours.

Figure — Local vs global minima - maxima

The word local is doing all the work: it means "compared only to the neighbourhood", not the whole world.

Step 2 — The slope: what "downhill" and "uphill" actually mean

WHAT. At every point on the curve you can lay a straight ruler that just kisses the curve — the tangent line. Its steepness is called the slope. We write it $f^{'} (x)$ (read "f-prime of x") — the derivative, which is just the number telling you how tilted the ruler is at that spot.

WHY THIS TOOL. Why bother with slope at all? Because "am I in a valley?" is really the question "is the ground flat right here?" — and slope is exactly the number that is positive when the ground rises to the right, negative when it falls, and zero when it is flat. No other single number answers "which way is downhill" so directly.

PICTURE. In the figure, three rulers are drawn. On the left of the valley the ruler tilts down to the right → slope negative (plum). On the right of the valley it tilts up → slope positive (orange). Right at the bottom it lies flat → slope zero (teal).

Step 3 — The necessary condition: bottoms and tops are flat

WHAT. At any local minimum or maximum of a smooth curve, the slope must be zero: $f^{'} (x^{*}) = 0$ Points where this happens are called critical points.

WHY. Here is the airtight reason, drawn as a picture. Suppose $x^{*}$ were a valley bottom but the slope there was not zero — say it pointed downhill to the right. Then stepping a hair to the right would take you lower — but that contradicts " $x^{*}$ is the lowest point around". The only escape is that the slope points nowhere: it is flat. Same argument for a peak.

PICTURE. The figure shows the contradiction: a tilted ruler at a supposed bottom always offers a lower spot right next door (red arrow). Only the flat ruler survives.

Step 4 — Curving up vs curving down: the second derivative

WHAT. The second derivative $f^{''} (x)$ (read "f-double-prime") is the slope of the slope — it measures how the tilt itself is changing as you walk right.

WHY THIS TOOL. Step 3 gave us flat spots but couldn't tell valleys from peaks. The trick: at a valley the ruler swings from tilting down to tilting up as you cross — the slope is increasing. At a peak the slope is decreasing. "How is the slope changing?" is answered by the slope-of-the-slope, i.e. $f^{''}$ . That's precisely why this second tool enters.

$f^{''} (x^{*}) > 0$ → slope increasing → the curve cups upward (a bowl 🥣) → local minimum
$f^{''} (x^{*}) < 0$ → slope decreasing → the curve caps downward (a dome) → local maximum
$f^{''} (x^{*}) = 0$ → test is silent — could be a shelf; look closer

PICTURE. The figure overlays a green bowl (cups up, $f^{''} > 0$ ) and a plum dome (caps down, $f^{''} < 0$ ), with little arrows showing the slope swinging from $-$ to $+$ across the bowl's bottom.

Step 5 — A full 1D example, every case shown

WHAT. Take $f (x) = x^{3} - 3 x^{2} + 2$ on the interval $x \in [- 1, 3]$ and find every extremum.

WHY. To prove the machine works, we run all three tests: find flat spots (Step 3), classify them (Step 4), and — new here — check the endpoints, because a bounded road can have its lowest point at a wall, not at a flat spot.

Find the flat spots. The slope is $f^{'} (x) = 3 x^{2} - 6 x = 3 x (x - 2)$

$3 x (x - 2)$ — a product, zero when either factor is zero
so $x = 0$ or $x = 2$ are the critical points

Classify them. The slope-of-the-slope is $f^{''} (x) = 6 x - 6$

at $x = 0$ : $f^{''} (0) = - 6 < 0$ → dome → local maximum
at $x = 2$ : $f^{''} (2) = 6 > 0$ → bowl → local minimum

Check every corner (all cases). Evaluate heights at the two flat spots and the two walls: $f (- 1) = - 2, f (0) = 2, f (2) = - 2, f (3) = 2$

PICTURE. The figure marks: left wall (teal, height $- 2$ ), the dome at $x = 0$ (orange, height $2$ ), the bowl at $x = 2$ (teal, height $- 2$ ), right wall (orange, height $2$ ).

Step 6 — Climbing into 2D: slope becomes a gradient, curvature becomes a matrix

WHAT. Now the landscape is a real surface: two horizontal controls $x$ and $y$ , one height $f (x, y)$ . The single slope splits into two — one for each direction — bundled into the gradient: $\nabla f = \frac{\partial f}{\partial x} \frac{\partial f}{\partial y}$

$\nabla f$ (read "grad f") — an arrow pointing in the steepest-uphill direction
$\frac{\partial f}{\partial x}$ — the slope if you walk in the $x$ direction only (freezing $y$ )
$\frac{\partial f}{\partial y}$ — the slope walking in $y$ only

WHY. In 1D "flat" was one number = 0. On a surface you can be flat east–west yet tilted north–south, so flatness now means both slopes vanish: $\nabla f = 0$ (the zero arrow). Same logic as Step 3, just in two directions at once.

PICTURE. The figure shows a bowl surface (contour rings). Little arrows (the gradient) all point outward-uphill; at the very bottom the arrow shrinks to nothing — that's $\nabla f = 0$ .

To classify a 2D flat spot we need curvature in every direction at once — that is the Hessian, a table of second derivatives $H_{ij} = \partial^{2} f / \partial x_{i} \partial x_{j}$ . If it curves up in all directions (all its eigenvalues positive) it's a minimum; down in all → maximum; up some, down others → a saddle point (a mountain pass: valley one way, ridge the other). See Loss Landscape Visualization for how these look in real training.

Step 7 — Why ML is hard: many valleys, and saddles everywhere

WHAT. Take the bumpy surface $f (x, y) = x^{2} + y^{2} - 2 cos (2 π x) - 2 cos (2 π y)$ .

WHY. The bowl $x^{2} + y^{2}$ has one true bottom at the origin. The two cosine terms carpet it with ripples — each ripple carves an extra little valley. Now a downhill walker (Gradient Descent) that only ever steps down can roll into the nearest ripple-valley and stop, blind to the deep global bowl below.

The heights. At the origin all cosines are at their peak ( $cos 0 = 1$ ), so $f (0, 0) = 0 - 2 - 2 = - 4 (global minimum)$ Every other flat spot sits higher — a local minimum only.

PICTURE. The figure is a top-down contour map: the deep central well (teal) is the global minimum; the surrounding ring of shallow dimples (plum) are traps. A dotted orange path shows a descent starting far out and halting in a shallow dimple.

The one-picture summary

Everything on this page in a single frame: find flat spots ( $f^{'} = 0$ ), split them with curvature ( $f^{''}$ or the Hessian), and remember that "flat + bowl" only gives you a local winner — global needs comparing all heights, walls included.

Recall Feynman retelling — say it back in plain words

Imagine a hilly road drawn as a line. First I ask: where is the ground flat? — that's the slope being zero, the derivative $f^{'} = 0$ . Those flat spots are my only suspects for a bottom or a top. But flat alone is a trick — it could be a bowl, a dome, or just a pause. So I ask a second question: is the ground cupping up or capping down here? — the slope-of-the-slope, $f^{''}$ . Cupping up means a valley, capping down means a peak. To find the very lowest point on a fenced road, I also peek at both fences, because the true low might be against a wall. Then I compare all those heights and pick the smallest — that's the global minimum. In 2D the single slope becomes an arrow (the gradient) that must shrink to zero, and curvature becomes a whole table (the Hessian) that must cup up in every direction. Real machine-learning landscapes are covered in ripples and mountain-passes, so a plain downhill walker can get stuck — which is why we give it momentum and clever step sizes to keep moving.

Recall Quick self-check

What condition makes $x^{*}$ a critical point? ::: The slope is flat there: $f^{'} (x^{*}) = 0$ (in 2D, $\nabla f (x^{*}) = 0$ ). Flat spot with $f^{''} (x^{*}) > 0$ is a…? ::: Local minimum (the curve cups upward like a bowl). Why must you check endpoints on a bounded interval? ::: The global extremum can sit against a wall, where the slope need not be zero. What is a saddle point? ::: A flat spot that curves up in some directions and down in others — a mountain pass, neither min nor max.