Critical points and saddle points: Foundations

Before you can read the parent note, you need to own every symbol it throws at you. We build each one from a picture, in the order that lets the next one make sense.

1. A function $f$ — the landscape itself

The picture: think of a hilly terrain. Your horizontal position needs two numbers (east–west, north–south) → that is $R^{2}$ . The height of the ground at that position is one number → that is $R$ . So a terrain is exactly $f : R^{2} \to R$ .

$R$ = "all real numbers" — every point on an infinite number line.
$R^{n}$ = "a list of $n$ real numbers" — a position needing $n$ coordinates.

Why the topic needs it. In machine learning $f$ is the loss — a number saying how wrong the model is. The $n$ inputs are the model's tunable parameters. We want the position of lowest height (least wrong). See Loss Landscape Visualization.

Figure — Critical points and saddle points

2. A point $x$ and the star $x^{*}$

The picture: on the terrain, $x$ is where you are standing (a pin on the map — exactly the orange dot from §1's figure). Its two numbers are how far east and how far north. $x^{*}$ is a marked pin — the pass or valley we are investigating.

Why the topic needs it. We cannot write "the flat point" a million different ways for a million-parameter network, so we bundle all coordinates into one bold letter and star the interesting one.

3. A one-input warm-up — the derivative

Why the limit? We want the slope at a single instant, not averaged over a chunk. The gap $h$ is the chunk width; sending $h \to 0$ (the symbol $lim$ means "the value this creeps toward") collapses the chunk to a point. That is the only tool that gives an instantaneous rate — see Taylor Series.

Sign story (all cases):

$f^{'} (x) > 0$ → ground rises to the right (uphill).
$f^{'} (x) < 0$ → ground falls to the right (downhill).
$f^{'} (x) = 0$ → flat — the ground momentarily neither rises nor falls. This is the flatness the whole topic hunts for.

4. Slope in every direction — partial derivatives and the gradient

Back to the full world: $f : R^{n} \to R$ with a bold input $x$ . With $n$ inputs there is no single slope: the ground can rise going east yet fall going north. We measure one direction at a time.

The picture: slice the terrain with a vertical wall running exactly east–west through your feet. On that wall's edge you see an ordinary 1-D curve. Its slope is $\frac{\partial f}{\partial x _{1}}$ .

Why an arrow? Each partial is one component of a total slope. Combined, they form the single direction of fastest increase — exactly what you'd need to climb fastest, or (with a minus sign) descend fastest.

5. Naming the three flat spots — local minimum, maximum, saddle

Before we measure curvature, let us pin down exactly what the words "minimum" and "maximum" mean, so "bowl" and "dome" are not just pictures.

The picture: "local" means look only close by. A valley floor may not be the deepest valley on Earth, but among its own neighbours it is lowest — that is enough to be a local minimum. The three shapes in the next section (bowl, dome, pass) are exactly these three cases.

6. Curvature — the second derivative and the Hessian

For many inputs (back to bold $x \in R^{n}$ ), curvature also depends on direction, so we need a whole grid of second derivatives.

Why $n \times n$ ? Every pair of directions has a bending relationship, and there are $n \times n$ ordered pairs. The diagonal ( $i = j$ ) holds pure up/down bending along each axis; off-diagonal entries hold twist. See Second-Order Optimization.

7. Reading the Hessian — quadratic forms and eigenvalues

The Hessian is a table of numbers; we need to squeeze one verdict out of it.

Formula Why

$d^{⊤} H d$ is the height change — from Taylor Take a tiny step $d$ from the critical point $x^{*}$ . The Taylor Series expansion (the multi-input version of "approximate a curve by a parabola") says $f (x^{*} + d) \approx start height f (x^{*}) + slope term \nabla f (x^{*})^{⊤} d + \frac{1}{2} curvature term d^{⊤} H d .$ What each piece is doing: the slope term is the first-order (straight-line) guess; the curvature term is the second-order correction that bends that line into a parabola. Why the middle term dies: at a critical point $\nabla f (x^{*}) = 0$ , so the slope term vanishes entirely. The leading surviving piece of the height change is then $f (x^{*} + d) - f (x^{*}) \approx \frac{1}{2} d^{⊤} H d .$ That is precisely why the sign of $d^{⊤} H d$ decides up-versus-down: positive → the step raises height, negative → it lowers height. The $\frac{1}{2}$ is a fixed positive factor, so it never changes the sign — it only scales the size. This is why $H$ "projects curvature along $d$ ": sandwiching $H$ between $d$ and $d$ reads off the bending in exactly that one direction.

Why scan all $d$ ? If some direction gives a negative value (an escape route downhill) while another gives positive, that mix is the saddle signature.

Why eigenvalues instead of the raw table? They are the cleanest summary of curvature — one number per independent direction. Their signs give the whole verdict:

All $λ_{i} > 0$	curves up everywhere	minimum (bowl)
All $λ_{i} < 0$	curves down everywhere	maximum (dome)
Mixed signs	up some ways, down others	saddle (pass)
Some $λ_{i} = 0$	flat in a direction	degenerate — look higher

This is the engine of the classification table in the parent note. Bowls are the friends we chase in Convex Optimization; saddles are the traps escaped by Momentum and Adam.

8. How it all feeds the topic

The left branch (slopes → gradient) finds where the flat spots are. The right branch (bending → Hessian → eigenvalues) tells us which kind of flat spot. The topic is exactly these two branches meeting at the verdict box.

Equipment checklist

Test yourself — say each answer aloud before revealing.

What does

f : R^{n} \to R

mean in plain words?

A machine taking a position of

n

numbers and returning one height number.

What is a bold

x

versus a plain

x_{i}

Bold

x

is the whole position vector (all coordinates);

x_{i}

is a single coordinate (a plain number).

Why does §3 use a plain scalar

x

instead of bold

x

It temporarily shrinks to one input (

f : R \to R

) to define the derivative without clutter; §4 returns to the bold

R^{n}

world.

Why does the derivative use a limit

h \to 0

To get the slope at a single instant/point instead of averaged over a finite chunk.

What does the curly

\partial

signal?

A partial derivative — the slope along one axis while all other inputs are frozen.

What is

\nabla f

and which way does it point?

The gradient, a vector bundling every partial slope; it points toward steepest ascent.

What does

\nabla f (x^{*}) = 0

say geometrically?

The ground is flat in every direction at

x^{*}

— a critical point.

What is the formal definition of a local minimum?

f (x^{*}) \leq f (x)

for all

x

in a small neighbourhood of

x^{*}

Why isn't a flat spot automatically a minimum?

It could equally be a maximum or a saddle; slope alone can't distinguish them.

What does the vertical bar

_{x^{*}}

mean?

"Evaluate this expression at the specific point

x^{*}

" after computing it.

What is the difference between

H

and

H_{ij}

Bold

H

is the whole Hessian matrix;

H_{ij}

is the single number in row

i

, column

j

, so

H = (H_{ij})

Why is the Hessian symmetric, and why does it matter?

Mixed partials are equal for smooth

f

(

H_{ij} = H_{j i}

), which guarantees real eigenvalues so their signs are meaningful.

Where does the quadratic form

\frac{1}{2} d^{⊤} H d

come from?

It is the surviving second-order term of the Taylor expansion once the gradient (first-order) term vanishes at a critical point.

What do the signs of

H

's eigenvalues decide?

All positive → minimum, all negative → maximum, mixed → saddle, any zero → degenerate.

Recall Quick self-check

If you can answer every line above without peeking, you are ready for the parent note's derivations and the Hessian classification table.