Critical points and saddle points: Exercises — AI-ML deep dive

This page is a self-test ladder. Each rung is harder than the last: L1 Recognition → L2 Application → L3 Analysis → L4 Synthesis → L5 Mastery. Read a problem, try it on paper, THEN open the collapsible solution. Every symbol you need was built in the parent note; if a step feels fast, revisit it there.

Quick reminders in one place:

For a $2 \times 2$ Hessian $H = (a b b c)$ there is a fast shortcut instead of solving the eigenvalue equation:

Level 1 — Recognition

Goal: read off critical points and name their type from the raw definitions.

Exercise 1.1 (L1)

Find all critical points of $f (x) = x^{2} - 6 x + 5$ and classify using $f^{''}$ .

Recall Solution 1.1

Step 1 — gradient (here just $f^{'}$ ). WHAT: set the slope to zero. WHY: a critical point is where the tangent line is flat. $f^{'} (x) = 2 x - 6 = 0 ⟹ x^{*} = 3.$ Step 2 — curvature. $f^{''} (x) = 2 > 0$ everywhere. A positive second derivative means the curve bends upward like a valley. Verdict: $x^{*} = 3$ is a local (and global) minimum. Value: $f (3) = 9 - 18 + 5 = - 4$ .

Exercise 1.2 (L1)

For $f (x, y) = x^{2} + y^{2}$ , write the gradient, find the critical point, and state its type by inspection.

Recall Solution 1.2

$\nabla f = (2 x, 2 y) = (0, 0) ⟹ (x^{*}, y^{*}) = (0, 0)$ . Hessian $H = (2002)$ , eigenvalues $2, 2$ — both positive. Verdict: minimum (a round bowl). $f (0, 0) = 0$ .

Exercise 1.3 (L1)

Which of these Hessians (each already evaluated at a critical point) is a minimum, a maximum, or a saddle? $H_{A} = (3001), H_{B} = (- 2 0 0 - 5), H_{C} = (40 0 - 1)$

Recall Solution 1.3

Read the diagonal directly since each is already diagonal — the diagonal entries are the eigenvalues (curvatures along the two axes).

$H_{A}$ : eigenvalues $3, 1$ both $> 0$ → minimum.
$H_{B}$ : eigenvalues $- 2, - 5$ both $< 0$ → maximum.
$H_{C}$ : eigenvalues $4, - 1$ mixed → saddle. Cross-check with $D = det$ : $D_{A} = 3 > 0$ , $D_{B} = 10 > 0$ , $D_{C} = - 4 < 0$ . Consistent.

Level 2 — Application

Goal: run the full pipeline — gradient, solve, Hessian, classify — on real functions.

Exercise 2.1 (L2)

Classify the critical point of $f (x, y) = x^{2} + x y + y^{2}$ .

Recall Solution 2.1

Gradient. $\nabla f = (2 x + y, x + 2 y) = (0, 0)$ . Solving: from the first, $y = - 2 x$ ; substitute into the second, $x + 2 (- 2 x) = - 3 x = 0 \Rightarrow x = 0$ , so $y = 0$ . Critical point $(0, 0)$ . Hessian. $H = (2112)$ . Test. $a = 2 > 0$ , $D = (2) (2) - 1^{2} = 3 > 0$ . Both positive → minimum. (Eigenvalues, if you want them: $λ = 2 \pm 1 = 3, 1$ , both positive. Same verdict.)

Exercise 2.2 (L2)

Find and classify the critical point of $f (x, y) = x^{2} - 4 x y + y^{2}$ .

Recall Solution 2.2

Gradient. $\nabla f = (2 x - 4 y, - 4 x + 2 y) = (0, 0)$ . From the first, $x = 2 y$ ; sub into second: $- 8 y + 2 y = - 6 y = 0 \Rightarrow y = 0, x = 0$ . Point $(0, 0)$ . Hessian. $H = (2 - 4 - 4 2)$ . Test. $D = (2) (2) - (- 4)^{2} = 4 - 16 = - 12 < 0$ → saddle. Geometry: along $y = x$ , $f = x^{2} - 4 x^{2} + x^{2} = - 2 x^{2}$ (curves down); along $y = - x$ , $f = x^{2} + 4 x^{2} + x^{2} = 6 x^{2}$ (curves up). Down in one direction, up in another — a saddle. See figure below.

Figure — Critical points and saddle points

Exercise 2.3 (L2)

$f (x) = x^{3} - 3 x$ . Find all critical points and classify each with the second-derivative test.

Recall Solution 2.3

Slope. $f^{'} (x) = 3 x^{2} - 3 = 0 \Rightarrow x^{2} = 1 \Rightarrow x = - 1, x = + 1$ . Two critical points. Curvature. $f^{''} (x) = 6 x$ .

At $x = - 1$ : $f^{''} (- 1) = - 6 < 0$ → local maximum, value $f (- 1) = - 1 + 3 = 2$ .
At $x = + 1$ : $f^{''} (+ 1) = 6 > 0$ → local minimum, value $f (1) = 1 - 3 = - 2$ . This is the classic cubic wiggle: up to a hill at $x = - 1$ , down to a valley at $x = 1$ .

Level 3 — Analysis

Goal: handle degenerate cases, multiple critical points, and reason about why a test fails.

Exercise 3.1 (L3)

$f (x) = x^{4}$ . Show the second-derivative test fails at the critical point, then determine the true type from first principles.

Recall Solution 3.1

Critical point. $f^{'} (x) = 4 x^{3} = 0 \Rightarrow x^{*} = 0$ . Second-derivative test. $f^{''} (x) = 12 x^{2}$ , so $f^{''} (0) = 0$ . WHAT this means: the quadratic term in the Taylor expansion vanishes, so the bowl/dome test is blind here — degenerate. Go higher. Near $0$ , $f (0 + h) = h^{4} \geq 0$ for every $h$ , with equality only at $h = 0$ . Since moving either way makes $f$ strictly larger, $x^{*} = 0$ is a strict local (and global) minimum — the curve is just extremely flat at the bottom. Contrast with $x^{3}$ (parent Example 1): there $f (0 + h) = h^{3}$ changes sign with $h$ , so that was an inflection, not an extremum. The parity of the lowest non-zero power decides: even → extremum, odd → inflection.

Exercise 3.2 (L3)

$f (x, y) = x^{2}$ (a function that ignores $y$ ). Find all critical points and explain the degenerate direction.

Recall Solution 3.2

Gradient. $\nabla f = (2 x, 0)$ . This is $0$ whenever $x = 0$ , for any $y$ . So the entire line ${(0, y) : y \in R}$ is critical — an infinite ridge, not an isolated point. Hessian. $H = (2000)$ , eigenvalues $2$ and $0$ . The zero eigenvalue is the $y$ -direction: $f$ is perfectly flat along $y$ , so the test is degenerate there. Because the other eigenvalue is positive and $f$ never decreases in any direction, every point on the ridge is a (non-strict) minimum — a flat-bottomed trough (a "valley floor"). This is exactly the kind of flat direction that plagues ill-conditioned loss landscapes.

Exercise 3.3 (L3)

$f (x, y) = x^{3} - 3 x y^{2}$ ("monkey saddle"). Find the critical point and show the standard $2 \times 2$ test is inconclusive.

Recall Solution 3.3

Gradient. $\frac{\partial f}{\partial x} = 3 x^{2} - 3 y^{2}$ , $\frac{\partial f}{\partial y} = - 6 x y$ . Setting both to zero: $- 6 x y = 0$ forces $x = 0$ or $y = 0$ ; either way $3 x^{2} - 3 y^{2} = 0$ then forces the other to be $0$ . Critical point: $(0, 0)$ . Hessian at origin. $\frac{\partial ^{2} f}{\partial x ^{2}} = 6 x = 0$ , $\frac{\partial ^{2} f}{\partial y ^{2}} = - 6 x = 0$ , $\frac{\partial ^{2} f}{\partial x \partial y} = - 6 y = 0$ . So $H = (0000)$ — the zero matrix, all eigenvalues $0$ : fully degenerate. True shape (higher order). Along a ray $y = 0$ : $f = x^{3}$ — up on one side, down on the other. Three "up" valleys and three "down" ridges alternate as you circle the origin (room for two legs and a tail — hence "monkey saddle"). It is a genuine saddle-type point that no second-order test can detect.

Level 4 — Synthesis

Goal: combine curvature, eigenvectors, and the optimization picture.

Exercise 4.1 (L4)

At a critical point of a loss $L$ you measure the Hessian $H = (50 0 - 3) .$ (a) Classify the point. (b) Give the exact direction $v$ (unit vector) in which you should perturb to decrease the loss fastest, and (c) estimate the loss change for a step of size $ϵ = 0.1$ along that direction using the quadratic model.

Recall Solution 4.1

(a) Eigenvalues $5$ and $- 3$ → mixed → saddle. (b) The most negative eigenvalue is $λ_{m i n} = - 3$ , whose eigenvector is the $y$ -axis, $v = (0, 1)$ . Loss curves downward fastest there, so that is the escape direction. (See Second-Order Optimization for why the min-eigenvalue direction is the escape route.) (c) Since $\nabla L = 0$ at a critical point, the quadratic model gives $Δ L \approx \frac{1}{2} ϵ^{2} v^{⊤} Hv = \frac{1}{2} (0.1)^{2} (- 3) = - 0.015.$ The loss drops by about $0.015$ — a small but definite decrease, confirming the point is not a minimum. This is precisely how noise in SGD/Adam nudges an optimizer off saddles.

Exercise 4.2 (L4)

Build a function of two variables whose only critical point is a saddle with curvature $+ 8$ along the direction $(1, 1) / 2$ and $- 2$ along $(1, - 1) / 2$ . Give $f$ explicitly.

Recall Solution 4.2

Idea (synthesis via eigen-decomposition). We want a quadratic $f = \frac{1}{2} x^{⊤} Hx$ whose Hessian has eigenvalue $8$ on eigenvector $u_{1} = (1, 1) / 2$ and $- 2$ on $u_{2} = (1, - 1) / 2$ . Reconstruct $H = 8 u_{1} u_{1}^{⊤} + (- 2) u_{2} u_{2}^{⊤}$ . $u_{1} u_{1}^{⊤} = \frac{1}{2} (1111), u_{2} u_{2}^{⊤} = \frac{1}{2} (1 - 1 - 1 1) .$ $H = 8 \cdot \frac{1}{2} (1111) - 2 \cdot \frac{1}{2} (1 - 1 - 1 1) = (4444) + (- 1 1 1 - 1) = (3553) .$ Function. $f (x, y) = \frac{1}{2} x^{⊤} Hx = \frac{1}{2} (3 x^{2} + 10 x y + 3 y^{2}) = \frac{3}{2} x^{2} + 5 x y + \frac{3}{2} y^{2} .$ Check. $\nabla f = (3 x + 5 y, 5 x + 3 y) = 0$ only at $(0, 0)$ ; Hessian $(3553)$ , $D = 9 - 25 = - 16 < 0$ → saddle. Its eigenvalues are $3 \pm 5 = 8, - 2$ . Exactly as required. ✓

Level 5 — Mastery

Goal: reason about high dimensions, convexity, and the statistics of saddles.

Exercise 5.1 (L5)

A critical point in $R^{n}$ has a Hessian with eigenvalues that are each independently positive or negative with probability $1/2$ (a toy model). (a) What is the probability the point is a local minimum? (b) Evaluate for $n = 1, n = 10, n = 1000$ . (c) What does this say about neural network landscapes?

Recall Solution 5.1

(a) A local minimum needs all $n$ eigenvalues positive. With independent fair coins, $P (all +) = (1/2)^{n} = 2^{- n}$ . (b)

$n = 1$ : $2^{- 1} = 0.5$ .
$n = 10$ : $2^{- 10} = 1/1024 \approx 9.77 \times 1 0^{- 4}$ .
$n = 1000$ : $2^{- 1000} \approx 1 0^{- 301}$ — effectively zero. (c) As dimension grows, minima become exponentially rare; almost every critical point has at least one negative eigenvalue and is therefore a saddle. This matches the parent note's claim that saddles dominate high-dimensional loss landscapes, and it is good news: saddles are escapable, unlike bad local minima. See Second-Order Optimization.

Exercise 5.2 (L5)

Show that a convex twice-differentiable function has no saddle points: every critical point is a global minimum.

Recall Solution 5.2

Fact used. For a convex $f$ , the Hessian is positive semi-definite everywhere: $d^{⊤} Hd \geq 0$ for all $d$ , i.e. no eigenvalue is ever negative (see Convex Optimization). Argument. A saddle requires at least one negative eigenvalue (a direction of downward curvature). Convexity forbids negative eigenvalues, so no saddle can exist. Any critical point has $\nabla f = 0$ ; by convexity the first-order Taylor bound $f (y) \geq f (x^{*}) + \nabla f (x^{*})^{⊤} (y - x^{*}) = f (x^{*})$ holds for all $y$ . Hence $x^{*}$ is a global minimum. This is why convex losses (linear/logistic regression) never have the saddle problem — but real deep nets are non-convex.

Exercise 5.3 (L5)

For $f (x, y) = x^{4} + y^{4} - 4 x y$ , find all critical points and classify each. (This mixes multiple critical points with a non-trivial off-diagonal Hessian.)

Recall Solution 5.3

Gradient. $\frac{\partial f}{\partial x} = 4 x^{3} - 4 y = 0 \Rightarrow y = x^{3}$ ; $\frac{\partial f}{\partial y} = 4 y^{3} - 4 x = 0 \Rightarrow x = y^{3}$ . Substitute $y = x^{3}$ into $x = y^{3}$ : $x = (x^{3})^{3} = x^{9} \Rightarrow x^{9} - x = 0 \Rightarrow x (x^{8} - 1) = 0$ . Real solutions $x = 0, x = 1, x = - 1$ , giving $y = x^{3} = 0, 1, - 1$ . Critical points: $(0, 0), (1, 1), (- 1, - 1)$ . Hessian. $H = (12 x^{2} - 4 - 4 12 y^{2})$ .

At $(0, 0)$ : $H = (0 - 4 - 4 0)$ , $D = 0 - 16 = - 16 < 0$ → saddle.
At $(1, 1)$ : $H = (12 - 4 - 4 12)$ , $a = 12 > 0$ , $D = 144 - 16 = 128 > 0$ → minimum, $f (1, 1) = 1 + 1 - 4 = - 2$ .
At $(- 1, - 1)$ : $H = (12 - 4 - 4 12)$ (same), → minimum, $f (- 1, - 1) = 1 + 1 - 4 = - 2$ . Picture: two symmetric valleys at $(\pm 1, \pm 1)$ separated by a saddle at the origin — a miniature two-basin loss landscape.

Recall Self-check: can you state the whole pipeline from memory?

Steps to classify any critical point ::: (1) solve $\nabla f = 0$ ; (2) build $H$ ; (3) eigenvalue signs — all $+$ min, all $-$ max, mixed saddle, any $0$ degenerate (go higher order). The $2 \times 2$ shortcut ::: $D = a c - b^{2}$ ; $D < 0$ saddle; $D > 0$ with $a > 0$ min, $a < 0$ max; $D = 0$ degenerate. Why saddles dominate high-D ::: a minimum needs all $n$ eigenvalues positive, probability $\sim 2^{- n}$ , vanishing as $n$ grows.