The Hessian matrix: Question bank — AI-ML deep dive

The vocabulary you need is all built in the parent: the Hessian $H$ is the matrix of second partial derivatives; a critical point is where the gradient $\nabla f = 0$ ; eigenvalues $λ_{i}$ measure curvature along special directions. We only reason here — no heavy computation.

Definition Three things we will lean on

Schwarz's theorem (equality of mixed partials): if the second partial derivatives of $f$ are continuous, then the order of differentiation does not matter, $f_{x y} = f_{y x}$ . This is why the Hessian is symmetric.
An eigenvector $q_{i}$ of $H$ is a special direction that $H$ merely stretches (does not rotate), and its eigenvalue $λ_{i}$ is the stretch factor — here the curvature along $q_{i}$ . For symmetric $H$ these $q_{i}$ are mutually perpendicular (an orthonormal eigenbasis).
The trace of $H$ , written $tr (H)$ , is just the sum of its diagonal entries — and for any square matrix that equals the sum of the eigenvalues, $tr (H) = λ_{1} + λ_{2}$ .
The condition number of a symmetric positive-definite $H$ is the ratio $κ = λ_{m a x} / λ_{m i n}$ — how much steeper the steepest direction is than the flattest. Big $κ$ = long thin valley.

The three pictures below anchor the three verdicts (s01), the diagonal-sign check (s02), and how rescaling warps curvature (s03). Refer back to them as you answer.

True or false — justify

True or false: the Hessian is only defined at critical points.

False —

H

is the matrix of second partials and exists wherever

f

is twice-differentiable; only the classification test (min/max/saddle) requires you to be at a critical point.

True or false: a real Hessian always has real eigenvalues.

True (when second partials are continuous) — by Schwarz's theorem

H

is symmetric, and real symmetric matrices have real eigenvalues via the spectral theorem.

True or false: if

H

is positive definite at every point,

f

is convex.

True — a positive-definite Hessian everywhere means every direction curves upward everywhere, which is exactly the second-order characterization of a convex function.

True or false:

det H > 0

guarantees a local minimum.

False —

det H > 0

only means the eigenvalues share a sign; they could both be negative, giving a maximum (see figure s01, middle dome, and the diagonal-sign check in figure s02). You must also inspect a diagonal entry or the trace.

True or false: a saddle point can occur even if

det H > 0

False — a saddle needs eigenvalues of opposite signs, so

det H = λ_{1} λ_{2} < 0

. Positive determinant rules out a 2×2 saddle.

True or false: the gradient and the Hessian both point "uphill".

False — the gradient is a vector pointing uphill; the Hessian is a matrix of curvatures with no single direction. It bends the hill rather than pointing along it.

True or false: doubling every coordinate scale keeps the eigenvalues of

H

unchanged.

False — rescaling variables warps the curvature (figure s03): if

S

is the diagonal scaling matrix defining the change of variables

x = S u

(each new coordinate

u_{i}

is an old coordinate divided by its scale), then the Hessian in the new variables is

S^{⊤} H S

, whose eigenvalues change and so does the condition number — this is exactly why preconditioning matters in optimization.

True or false: if

f

has a zero eigenvalue at a critical point, it must be a degenerate saddle.

False — a zero eigenvalue makes the test inconclusive, not a saddle; the flat direction needs higher-order terms to resolve.

True or false: the Hessian of a purely linear function

f = a^{⊤} x

is the zero matrix.

True — all second partials of a linear function vanish, so

H = 0

; consistent with the fact that a plane has no curvature.

True or false: for a quadratic

f (x) = \frac{1}{2} x^{⊤} A x

with symmetric

A

, the Hessian equals

A

everywhere.

True — a quadratic's second derivatives are constant, so

H = A

at every point, which is why quadratics are the model Newton's method solves in one step.

Spot the error

"We found

\nabla f \neq = 0

here but

H

is positive definite, so it's a local minimum."

Error — the second-derivative test is only valid at critical points. Off a critical point the linear term dominates, so there is nothing to classify.

det H = 0

, so this critical point is a saddle."

Error —

det = 0

means at least one zero eigenvalue, which makes the test inconclusive, not a saddle. A saddle needs strictly opposite-sign eigenvalues.

H = [2152]

is a valid Hessian."

Error — for a function

f \in C^{2}

(second partials continuous) Schwarz's theorem forces

f_{x y} = f_{y x}

, so a Hessian must be symmetric; but

5 \neq = 1

off-diagonal, so this cannot be such a Hessian.

"Adam adapts each step using the Hessian, so it's a curvature method."

Error — Adam uses only first-order info (running averages of gradients and squared gradients). It never forms or approximates second derivatives.

"The trace of

H

is positive, therefore the point is a minimum."

Error — the trace

tr (H) = λ_{1} + λ_{2}

can be positive while one eigenvalue is negative (e.g.

5 + (- 1) = 4 > 0

yet

det = λ_{1} λ_{2} = - 5 < 0

is a saddle). Positivity of the sum does not force both signs positive.

"Newton's step is

x \leftarrow x - H \nabla f

Error — Newton's method uses the inverse Hessian:

x \leftarrow x - H^{- 1} \nabla f

. Multiplying by

H

instead of

H^{- 1}

has wrong units and diverges.

f_{x y}

and

f_{y x}

can differ, so the Hessian is generally non-symmetric."

Error — for

f \in C^{2}

(continuous second partials) Schwarz's theorem forces

f_{x y} = f_{y x}

, so

H

is symmetric. Only functions failing that regularity hypothesis can break it.

Why questions

Why does 1D need only one second-derivative number but

n

D needs a whole matrix?

In 1D there is a single direction to curve, but in

n

dimensions curvature depends on the direction you look and directions can couple, so you write

n^{2}

entries as a matrix — though Schwarz's symmetry (

H_{ij} = H_{j i}

) makes only

n (n + 1) /2

of them independent.

Why do we use eigenvalues rather than raw matrix entries to classify a point?

If we write a nudge

Δ x

in the eigenbasis as

\sum_{i} u_{i} q_{i}

— where each

q_{i}

is an eigenvector of

H

and

u_{i}

is the coordinate along it — the quadratic form decouples into

\frac{1}{2} \sum_{i} λ_{i} u_{i}^{2}

, so each eigenvalue

λ_{i}

is a clean curvature along its own axis and the signs of the

λ_{i}

directly give min/max/saddle.

Why does the linear (gradient) term disappear when classifying a critical point?

At a critical point

\nabla f = 0

, so the first-order term of the Taylor expansion vanishes and the Hessian term alone decides the local shape.

Why is the symmetry of the Hessian a "gift" for optimization?

Symmetry (guaranteed by Schwarz's theorem) gives real eigenvalues and an orthonormal eigenbasis of the

q_{i}

, which is exactly what lets us cleanly diagonalize the curvature and read off min/max/saddle.

Why does a large condition number

κ = λ_{m a x} / λ_{m i n}

slow down Gradient descent?

A large

κ

means a long thin valley (figure s03, right) where the gradient points mostly across the valley, so plain descent zig-zags instead of heading down the trough.

Why do deep-learning frameworks avoid computing the full Hessian?

For

n

parameters

H

n \times n

, which is astronomically large for millions of weights, so methods like L-BFGS build cheap low-rank curvature estimates from gradient differences instead.

Edge cases

What does the test say if both eigenvalues are zero?

Completely inconclusive — the surface is flat to second order in every direction, so only higher-order terms can decide the shape.

What is the Hessian of

f (x, y) = x^{3}

at the origin, and what does the test conclude?

H = [6 x 0 00]

evaluates to the zero matrix at the origin, so both eigenvalues are

0

and the test is inconclusive — indeed

(0, 0)

is an inflection, not a min/max.

H

is positive semi-definite (one zero, one positive eigenvalue) at a critical point, is it a minimum?

Not determined — the positive direction curves up but the flat direction is undecided, so it may be a minimum or a degenerate saddle depending on higher-order behaviour.

Can a function have

det H < 0

everywhere yet no critical point at all?

Yes —

det H

describes curvature independent of the gradient; if

\nabla f

never vanishes there is no point to classify even though the surface is everywhere saddle-shaped.

At a maximum, what are the signs of the eigenvalues and the determinant (2×2 case)?

Both eigenvalues negative, so the determinant

λ_{1} λ_{2} > 0

(positive) while a diagonal entry like

a < 0

(figure s02) — the positive determinant is why you must also check a diagonal sign to distinguish max from min.

Recall One-line summary of every trap

The Hessian exists everywhere but only classifies at critical points; $det > 0$ needs a diagonal-sign check; zero eigenvalue means inconclusive, never saddle; Adam is first-order; Newton uses $H^{- 1}$ .