The Hessian matrix: Foundations — AI-ML deep dive

Before you can read The Hessian matrix, a stream of new symbols will appear. This page introduces them one at a time, from zero, and nothing is used before it is defined and pictured. If you can read a graph, you can read this. By the last section you will meet the Hessian matrix itself — fully assembled from the atoms below.

1 · What is a function of many variables?

The picture. Think of a landscape. You stand at a spot given by two coordinates $(x, y)$ (east–west and north–south), and $f (x, y)$ is the altitude at your feet. The whole function is a hilly surface floating above the flat ground.

Why the topic needs it. The Hessian lives on such surfaces. It only makes sense when there are several input directions — that is exactly what $R^{n}$ gives us. A single number ( $n = 1$ ) has just one way to curve; more inputs mean more directions, and that is the whole reason the Hessian is a matrix.

2 · The partial derivative $\frac{\partial f}{\partial x}$

Why "freeze the others"? On a landscape, if you walk due east and hold your north–south position fixed, you feel a definite up-or-down slope. That slope is $\partial f / \partial x$ . Walk due north instead: $\partial f / \partial y$ . Each freezes the other so the question "how steep?" has one clear answer.

Why the topic needs it. Every entry of the gradient and every entry of the Hessian is a partial derivative. This is the atom of the whole subject.

3 · The gradient vector $\nabla f$

The picture. At every point on the landscape, $\nabla f$ is an arrow lying flat on the ground pointing in the direction of steepest uphill. Its length says how steep. Where the ground is flat (a peak, a pit, a saddle), the arrow shrinks to nothing: $\nabla f = 0$ .

Why the topic needs it. The Hessian starts working after the gradient. A critical point is where $\nabla f = 0$ — the flat spots. Only there does the Hessian get to decide "peak, pit, or saddle?". See The gradient vector for the full story.

4 · The second derivative — curvature in 1D

The picture. Imagine driving along a road profile:

$f^{''} > 0$ : the road bends upward — a valley shape (smile). $⌣$
$f^{''} < 0$ : the road bends downward — a hill shape (frown). $⌢$
$f^{''} = 0$ : no bending here — locally straight.

Why the topic needs it. The Hessian is literally "the second derivative for functions of many variables." Understand $f^{''}$ as a picture (smile vs frown) and every eigenvalue of it will read the same way.

5 · Second partial derivatives — and building the Hessian $H$

Now combine the two atoms of the last sections: partial (freeze the others) and second derivative (differentiate twice). In many variables you can differentiate twice in the same direction, or in two different directions.

Why a grid and not a single number? In 1D there is one curvature, $f^{''}$ . With several inputs, curvature depends on which direction you look, and directions can interact — the mixed partials capture that interaction. You need $n^{2}$ numbers, so a matrix.

6 · Vectors, length, transpose, and the "sandwich" $v^{⊤} H v$

How to read the sandwich. $v$ is the direction you choose to walk; $H$ is the curvature grid; the output number reports the curvature along that path. Positive ⇒ that path bends up (valley-like); negative ⇒ bends down (hill-like).

Intuition Why the tool: why a quadratic form, and where the

$\frac{1}{2}$ comes from Taylor's theorem says a smooth function near a point $x$ can be rebuilt from its derivatives. Write $Δ x$ (a vector, hence bold) for a small displacement — the step you take away from $x$ . Then, keeping terms up to second order, $f (x + Δ x) \approx height f (x) + slope term \nabla f (x)^{⊤} Δ x + curvature term \frac{1}{2} Δ x^{⊤} H Δ x .$ Where does the $\frac{1}{2}$ come from? It is inherited exactly from 1D. Taylor's rule builds the $k$ -th term by dividing the $k$ -th derivative by $k! = 1 \cdot 2 \dots k$ . The second-order term uses $2! = 2$ , so it carries a factor $\frac{1}{2 !} = \frac{1}{2}$ — this is why the 1D formula reads $\frac{1}{2} f^{''} (x) h^{2}$ , and the multivariable curvature term simply copies that same $\frac{1}{2}$ . (Concretely: for the plain parabola $f (t) = \frac{1}{2} c t^{2}$ , we have $f^{''} = c$ , and only the coefficient $\frac{1}{2} c$ reproduces the original — so the $\frac{1}{2}$ is forced.) The leftover remainder shrinks faster than $∥Δ x ∥^{2}$ and is dropped for small steps. The Hessian appears in the curvature term because the quadratic form is the smallest gadget that can bend differently in different directions, which a single number never could. This is the doorway to Taylor series (multivariable), Positive definite matrices and Convex functions.

7 · The determinant $det H$

Why the topic needs it. The fast $2 \times 2$ second-derivative test in the parent note is built on $det H$ plus the diagonal check. It replaces "find the eigenvalues" with "check one product and one sign."

8 · Eigenvalues $λ$ , eigenvectors, and the trace

The picture for the Hessian. Because $H$ is symmetric, its eigenvectors are the "natural axes" of the curved surface — the directions along which the bowl or saddle is aligned. Along each such axis the surface behaves like a simple 1D parabola, and its eigenvalue $λ$ is that parabola's $f^{''}$ :

$λ > 0$ ⇒ smile (valley) along that axis,
$λ < 0$ ⇒ frown (hill) along that axis.

Definition The trace, and its link to determinant and eigenvalues

The trace of a square matrix, written $tr H$ , is simply the sum of its diagonal entries: for $H = [a b b c]$ , $tr H = a + c$ . Two clean facts hold for a $2 \times 2$ symmetric matrix with eigenvalues $λ_{1}, λ_{2}$ : $tr H = λ_{1} + λ_{2} (sum), det H = λ_{1} λ_{2} (product) .$ Why these hold: the eigenvalues are the two roots of the equation $λ^{2} - (a + c) λ + (a c - b^{2}) = 0$ (the "characteristic equation" whose derivation belongs to Eigenvalues and eigenvectors). By the rule relating a quadratic's roots to its coefficients, the roots' sum is $a + c = tr H$ and their product is $a c - b^{2} = det H$ . This is exactly why, when $det H > 0$ (eigenvalues share a sign), the sign of $tr H$ — or equivalently of the diagonal entry $a$ — tells you which sign: positive ⇒ both up ⇒ minimum, negative ⇒ both down ⇒ maximum.

Prerequisite map

Everything flows into one place: The Hessian matrix. Once these atoms are solid, the parent note is just assembling them — and downstream tools Newton's method, Gradient descent and Convex functions all reuse the same ideas.

Equipment checklist

Try to answer each before revealing. If any stumps you, reread its section above.

What does

f : R^{2} \to R

take in and give out?

Two numbers

(x, y)

in, one number (a height) out — a surface over the plane.

What smoothness do we assume about

f

, and what does it buy us?

f \in C^{2}

(second partials exist and are continuous); this guarantees the Hessian exists AND is symmetric.

What does the curly

\partial

signal that the straight

d

does not?

That other variables exist and are being held frozen while we differentiate w.r.t. one.

In one word, what does the gradient

\nabla f

point toward?

Steepest-uphill.

What is the Hessian

H

, in one sentence?

The square grid of all second partial derivatives,

H_{ij} = \partial^{2} f / \partial x_{i} \partial x_{j}

— the multivariable second derivative.

What does the transpose

M^{⊤}

do, and what makes a matrix symmetric?

It mirrors the matrix across its diagonal (rows become columns); symmetric means

M = M^{⊤}

Why is

H

symmetric?

Because for continuous second partials the mixed partials are equal,

\partial^{2} f / \partial x \partial y = \partial^{2} f / \partial y \partial x

— both measure the same twist.

How do you compute the length

∥ v ∥

of a vector, and what is a unit vector?

By Pythagoras,

∥ v ∥ = v_{1}^{2} + v_{2}^{2}

; a unit vector has length exactly

1

Where does the

\frac{1}{2}

in the curvature term come from?

From Taylor's

1/ k!

rule at order

k = 2

1/2! = \frac{1}{2}

, exactly as in the 1D term

\frac{1}{2} f^{''} (x) h^{2}

Where does the second-derivative test become usable, and why?

Only at critical points (

\nabla f = 0

), because there the slope is flat and only curvature decides the shape.

Smile or frown: does

f^{''} > 0

describe a valley or a hill?

Valley (curves upward,

⌣

Why must

v

be a unit vector to read true directional curvature from

v^{⊤} H v

Because scaling

v

scales the value by the square of its length; only with

∥ v ∥ = 1

is the number the real curvature — otherwise only its sign is meaningful.

det H > 0

— is that a minimum? What extra check is needed?

Not yet;

det > 0

only means the curvatures share a sign. Check

a

(or the trace):

a > 0

⇒ min,

a < 0

⇒ max.

For a

2 \times 2

symmetric

H

, what are

det H

and

tr H

in terms of eigenvalues?

det H = λ_{1} λ_{2}

(product) and

tr H = λ_{1} + λ_{2}

(sum).

What geometric fact makes an eigenvector special for

H

H

only stretches it (by

λ

) without rotating it — it is a natural axis of the surface, and

λ

is the 1D curvature along it.