Foundations — Local vs global minima - maxima

This page assumes you know nothing beyond basic arithmetic. We will build every symbol the parent note Local vs global minima - maxima throws at you — one brick at a time, each earned before it is used.

1. What is a function $f$?

Picture a vending machine: press a button (the input $x$), out drops one snack (the output $f(x)$). Never two snacks, never zero — exactly one.

In the picture, the input $x$ runs left–right along the flat ground. The output $f(x)$ is the height of the curve above that spot. So a function is literally a landscape: input = where you stand, output = how high you are.

2. The symbol $\mathbb{R}$ and the arrow $f:{\mathbb{R}}^n\to \mathbb{R}$

Picture $\mathbb{R}$ as an endless ruler. Picture ${\mathbb{R}}^2$ as a floor tiled with coordinates. The arrow says: stand anywhere on the floor, the machine tells you one height.

3. Points, stars, and bold letters: $x$, $\mathbf{x}$, ${\mathbf{x}}^{\ast }$

Think of ${\mathbf{x}}^{\ast }$ as the little flag planted at the bottom of a valley.

4. Slope, the derivative $f^{\prime }(x)$, and "flat"

Before minima make sense, you must be able to see steepness.

Look at the figure: the green line just touches the curve at one point — that touching line is called the tangent, and its steepness is $f^{\prime }(x)$. Where the curve turns around at a valley or a peak, the tangent lies perfectly flat, so $f^{\prime }(x)=0$.

5. The second derivative $f^{\prime \prime }(x)$ — which way does it curve?

A flat spot could be a valley bottom, a hilltop, or a flat ledge. To tell them apart we need curvature.

6. Going multi-dimensional: the gradient $\nabla f$

On a floor ${\mathbb{R}}^n$ there isn't just one direction to step — there are many. So one slope number is not enough.

Picture standing on a hillside: the gradient is the arrow pointing the way water would not flow — straight up the steepest slope. Walk against it and you go downhill fastest. That is why Gradient Descent takes steps in the direction $-\nabla f$.

7. The Hessian $H$ — curvature in every direction

Just as $f^{\prime \prime }$ tested curvature in 1D, we need a curvature test in many dimensions.

The Hessian's job: tell you the shape of the bowl in every direction at once.

• Positive definite (all curvatures effectively "cup-like") → valley → local minimum.
• Negative definite (all "cap-like") → peak → local maximum.
• Mixed → a saddle: a cup one way, a cap the crossways direction (like a horse saddle or a Pringle).

8. The neighbourhood: $ϵ$, the norm $\Vert \cdot \Vert$, and $\forall$

The formal definition of local needs a way to say "nearby only."

So $f({\mathbf{x}}^{\ast })\le f(\mathbf{x})\forall \mathbf{x}$ with $\Vert \mathbf{x}-{\mathbf{x}}^{\ast }\Vert <ϵ$ reads: "our spot is lowest among everybody inside the tiny circle." That is a local minimum — lowest only nearby. Drop the circle (compare against all points) and you get a global minimum.

9. The whole-picture map

Everything on the left is a prerequisite brick; the topic sits at node H; and the payoff is Gradient Descent, later refined by Momentum and Nesterov and Adam Optimizer.

Equipment checklist

Cover the right side and answer aloud. If any stumps you, re-read its section above.