Critical points and saddle points

What Are Critical Points?

Why does the gradient vanish? The gradient $\nabla f$ points in the direction of stepest ascent. At a critical point, there's no direction that immediately increases or decreases $f$ —the function is locally "flat." This doesn't tell us whether we're at a maximum, minimum, or saddle; it just says the first-order behavior is zero.

Derivation from first principles:

For $f$ to have a local extremum at $\mathbf{x}^*$ , moving slightly in any direction $\mathbf{v}$ shouldn't change $f$ to first order: $f(\mathbf{x}^* + \epsilon \mathbf{v}) \approx f(\mathbf{x}^*) + \epsilon \nabla f(\mathbf{x}^*)^\top \mathbf{v}$

For this to be an extremum, the linear term must vanish for all $\mathbf{v}$ , which forces $\nabla f(\mathbf{x}^*) = \mathbf{0}$ .

Classifying Critical Points: The Hessian

Once we find $\nabla f(\mathbf{x}^*) = \mathbf{0}$ , we examine the second-order behavior via the Hessian matrix.

Why the Hessian? Taylor expansion around $\mathbf{x}^*$ gives: $f(\mathbf{x}^* + \mathbf{d}) \approx f(\mathbf{x}^*) + \nabla f(\mathbf{x}^*)^\top \mathbf{d} + \frac{1}{2} \mathbf{d}^\top \mathbf{H} \mathbf{d}$

Since $\nabla f(\mathbf{x}^*) = \mathbf{0}$ , the behavior is governed by the quadratic form $\mathbf{d}^\top \mathbf{H} \mathbf{d}$ . The sign of this form determines whether $f$ increases or decreases as we move away.

Why eigenvalues? The eigenvalues of $\mathbf{H}$ are the curvatures along the principal axes. If all are positive, the function curves upward in every direction (a bowl). If mixed, it curves up in some directions and down in others (a sadle).

Figure — Critical points and saddle points

Saddle Points in Deep Learning

Why sadles dominate in high-D:
For a random symmetric matrix (a rough model of $\mathbf{H}$ ), the probability of all $n$ eigenvalues having the same sign shrinks exponentially with $n$ . In $\mathbb{R}^{1000000}$ , the chance of a local minimum is astronomically small; sadles are the norm.

Implication for optimization: Gradient descent can get stuck near saddles where $\|\nabla f\| \approx 0$ but the point is not a minimum. Algorithms like Adam, SGD with momentum, and sadle-free Newton methods help escape saddles by adding noise or using curvature information.

Worked example Example 2: Two-Variable Sadle

$f(x, y) = x^2 - y^2$ (hyperbolic paraboloid)

Find critical point: $\nabla f = (2x, -2y) = (0, 0) \implies (x^*, y^*) = (0, 0)$ Compute Hessian: $\mathbf{H} = \begin{pmatrix} \frac{\partial^2 f}{\partial x^2} & \frac{\partial^2 f}{\partial x \partial y} \\ \frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial y^2} \end{pmatrix} = \begin{pmatrix} 2 & 0 \\ 0 & -2 \end{pmatrix}$

Eigenvalues: $\lambda_1 = 2 > 0$ , $\lambda_2 = -2 < 0$

Conclusion: $(0,0)$ is a saddle point. Along $x$ -axis ( $y=0$ ), $f(x,0) = x^2$ curves upward. Along $y$ -axis ( $x=0$ ), $f(0,y) = -y^2$ curves downward.

Why this step? The mixed-sign eigenvalues prove the quadratic form $\mathbf{d}^\top \mathbf{H} \mathbf{d} = 2d_x^2 - 2d_y^2$ is positive for some $\mathbf{d}$ and negative for others. This is the defining property of a saddle.

Worked example Example 3: Neural Network Loss

Consider a two-layer network with loss $L(\mathbf{w})$ in parameter space. Suppose gradient descent finds $\nabla L(\mathbf{w}^*) \approx \mathbf{0}$ and $\|\nabla L\|$ stops decreasing.

Check if stuck at sadle:

Compute or approximate Hessian eigenvalues (e.g., via Lanczos or power iteration)
If smallest eigenvalue $\lambda_{\min} < 0$ , there's a direction $\mathbf{v}$ where loss decreases: $\mathbf{v}$ is the eigenvector for $\lambda_{\min}$
Perturb $\mathbf{w}^* \to \mathbf{w}^* + \epsilon \mathbf{v}$ to escape Why this step? Negative eigenvalue means the loss surface curves downward in direction $\mathbf{v}$ . Moving that way decreases loss, proving $\mathbf{w}^*$ is not a minimum. Modern optimizers add noise (SGD's minibatch variance) or momentum to naturally perturb out of saddles.

Common Mistakes

Common mistake Mistake 2: "Sadle points trap optimization"

Why it feels right: If gradient descent reaches a saddle where $\nabla f = 0$ , the update $\mathbf{w}_{t+1} = \mathbf{w}_t - \alpha \nabla f(\mathbf{w}_t)$ yields $\mathbf{w}_{t+1} = \mathbf{w}_t$ —stuck!

The fix: In practice, saddles are unstable equilibria. Numerical noise, finite precision, or stochastic gradients (from minibatches) perturb the iterate slightly. Once perturbed, the negative eigenvalue direction pulls the optimizer away. Saddles slow convergence but rarely trap completely.

Research insight: Ge et al. (2015) proved that gradient descent with random initialization escapes strict saddles (where $\lambda_{\min} < -c < 0$ ) in polynomial time.

Active Recall

Recall Explain to a 12-year-old

Imagine you're playing a video game where you're a marble rolling on a bumpy surface, and you want to find the lowest valley. A critical point is anywhere the ground is perfectly flat—you stop rolling because there's no slope. But "flat" doesn't always mean you're at the bottom of a valley! You could be:

At the bottom of a bowl (minimum)—you found a low point, yay!
At the top of a hill (maximum)—you're at a high point, bad if you want low.
On a saddle (like horse sadle)—flat in the middle, but if you roll forward you go down, and if you roll sideways you go up. To figure out which, you check the curvature—does the ground curve up like a smile (minimum), down like a frown (maximum), or mixed (saddle)? In math, the Hessian matrix tells us the curvature in all directions at once.

Connections

Gradient Descent: Critical points are where gradient descent converges (or gets stuck)
Second-Order Optimization: Newton's method uses Hessian to jump toward minima, avoiding saddles
Loss Landscape Visualization: Tools like loss surface plots reveal the density of sadles in deep nets
Convex Optimization: In convex functions, every critical point is a global minimum—sadles impossible
Momentum and Adam: These optimizers help escape saddles via velocity/adaptive learning rates
Eigenvalues and Eigenvectors: Hessian's eigenvalues/vectors describe principal curvatures
Taylor Series: Second-order Taylor expansion motivates the Hessian classification

#flashcards/ai-ml

What is a critical point of a function $f$ ? :: A point $\mathbf{x}^*$ where the gradient vanishes: $\nabla f(\mathbf{x}^*) = \mathbf{0}$ . All partial derivatives are zero.

What does the Hessian matrix represent?

The matrix of second partial derivatives

H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}

, capturing the curvature of

f

at a point.

How do you classify a critical point using the Hessian's eigenvalues?

All positive → local minimum; all negative → local maximum; mixed signs → saddle point; some zero → degenerate (need higher derivatives).

Why are saddle points common in high-dimensional neural network optimization?

In high dimensions, the probability that all

n

eigenvalues have the same sign is exponentially small. Most critical points have mixed-sign eigenvalues, making them sadles.

What is the quadratic form approximation near a critical point?

f(\mathbf{x}^* + \mathbf{d}) \approx f(\mathbf{x}^*) + \frac{1}{2} \mathbf{d}^\top \mathbf{H} \mathbf{d}

, where the gradient term vanishes at

\mathbf{x}^*

How can you escape a saddle point in optimization?

Perturb along the eigenvector corresponding to the negative eigenvalue, or rely on noise from stochastic gradients and momentum to naturally escape.

What does a positive definite Hessian indicate?

The critical point is a local minimum—the function curves upward in all directions (all eigenvalues

> 0

What does an indefinite Hessian indicate? :: The critical point is a saddle—the function curves upward in some directions and downward in others (mixed-sign eigenvalues).

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Critical points aur saddle points kya hain? Jab ap kisi function ka minimum dhundh rahe hain—jaise neural network ki loss—toh apko pehle wo jagah chahiye jahan gradient zero ho jaye, matlab $\nabla f = \mathbf{0}$ . Ye critical point kehlata hai. Lekin yahan problem hai: zero gradient ka matlab yeh nahi ki apko minimum mil gaya! Aap maximum par bhi ho sakte hain (jaise pahad ki choti), ya saddle point par—jahan ek direction mein surface neeche jata hai aur dusre mein upar (jaise ghode ki seat).

Classification kaise karein? Hessian matrix dekho—ye second derivatives ka matrix hai jo curvature bata hai. Agar Hessian ke sare eigenvalues positive hain, toh aap minimum mein hain (bowl shape). Agar sare negative, toh maximum (inverted bowl). Agar kuch positive aur kuch negative hain, toh sadle point hai. Deep learning mein problem yeh hai ki bohot saare parameters hone ke wajah se (millions!), critical points mein se zyada tar saddles hote hain, minima bahut rare hain.

Optimization mein kya hota hai? Gradient descent sadle points par slow ho jata hai kyunki gradient zero ke pas hota hai. Lekin modern optimizers (Adam, momentum, SGD with noise) sadles se escape kar lete hain—negative eigenvalue wali direction mein naturally perturb ho jate hain. Isliye practice mein saddleszyada problem nahi bante, bas thoda time lagta hai unse nikalne mein.

Real-world analogy: Socho ek hiker andhera mein pahaad par chal raha hai. Jahan zameen bilkul flat lage (zero slope = zero gradient), wahan ruk jata hai. Par flat zameen teen tarah ki ho sakti hai: valley (minimum—good!), hiltop (maximum—bad), ya saddle (ek taraf down, ek taraf up—confusing!). Hessian eigenvalues se aap batate hain kis tarah ki flatness hai, bina as-paas dekhe.

What Are Critical Points?

Classifying Critical Points: The Hessian

Saddle Points in Deep Learning

Common Mistakes

Active Recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Calculus & Optimization Basics

Connections