The Jacobian matrix: Question bank — AI-ML deep dive

Before we start, two words we lean on constantly. Output = one of the numbers the function spits out ( $f_{i}$ ). Input = one of the numbers you feed in ( $x_{j}$ ). The whole game is keeping those two straight, because the Jacobian is literally a table indexed by (output, input).

True or false — justify

For

f : R^{n} \to R^{m}

, the Jacobian is

n \times m

False. It is

m \times n

— rows are outputs, columns are inputs — because

J h

must eat an input direction

h \in R^{n}

and return an output change in

R^{m}

(m \times n) (n \times 1) = (m \times 1)

The gradient and the Jacobian are the same object.

False. They only coincide (up to a transpose) when the output is a single scalar; the gradient

\nabla f

is a column, the Jacobian for

m = 1

is its row transpose

(\nabla f)^{⊤}

. For

m > 1

there is no single gradient at all.

Every Jacobian has a determinant.

False. A determinant needs a square matrix, so

det J

exists only when

m = n

. Most ML layers map

R^{n} \to R^{m}

with

m \neq = n

, so no determinant exists for them.

The chain rule for Jacobians,

J_{g \circ f} = J_{g} J_{f}

, is order-independent because scalar chain rule is.

False. Matrix multiplication does not commute; only the order "outer Jacobian on the left" makes the shapes line up. Writing

J_{f} J_{g}

is generally undefined or wrong.

The Jacobian is defined only at a specific point

x

, not once for the whole function.

True. Each entry

\partial f_{i} / \partial x_{j}

is a function of

x

, so

J

is a field of matrices; you must evaluate it at a point to get concrete numbers (like

f^{'} (x)

varying along a curve).

det J = 0

at a point, the map

f

locally collapses area/volume to zero there.

True.

∣ det J ∣

is the local volume-scaling factor, so a zero determinant means a tiny patch is flattened onto something lower-dimensional — the map is not locally invertible there.

A diagonal Jacobian means the map is linear.

False. Diagonal only means each output depends on one input (no cross-coupling), e.g. elementwise

σ (z)

. The map can still be highly nonlinear; the entries

σ^{'} (z_{i})

just change from point to point.

The linear approximation

f (x + h) \approx f (x) + J h

is exact when

f

is affine.

True. For

f (x) = A x + b

we get

J = A

constant, and

A (x + h) + b = (A x + b) + A h

exactly — no higher-order terms exist to drop.

Spot the error

"Row

j

of the Jacobian is the gradient of input

x_{j}

Rows are indexed by outputs, not inputs, and gradients belong to outputs: row

i

(\nabla f_{i})^{⊤}

. Inputs are the columns; "gradient of an input" is meaningless.

"For

z = W x

the Jacobian is

W^{⊤}

because we differentiate."

It is

W

, not

W^{⊤}

. Since

z_{i} = \sum_{j} W_{ij} x_{j}

, we get

\partial z_{i} / \partial x_{j} = W_{ij}

, which reproduces

W

exactly with no transpose.

"Since

a_{i} = σ (z_{i})

is nonlinear, its Jacobian is a full dense matrix."

Each

a_{i}

depends only on its own

z_{i}

, so all off-diagonal partials

\partial a_{i} / \partial z_{j}

(

i \neq = j

) are zero. The Jacobian is

diag (σ^{'} (z_{1}), \dots)

— diagonal, not dense.

"To linearize, use

f (x) + h J

Shapes are wrong and the order is backward: it must be

f (x) + J h

, a matrix

(m \times n)

times a column

(n \times 1)

. Writing

h J

tries to multiply

(1 \times n) (m \times n)

, which is undefined.

"In backprop we multiply Jacobians front-to-back as

J_{1} J_{2} \dots J_{L}

following the forward order."

The chain rule stacks outer on the left, so the composition orders them last-layer-first:

J_{L} \dots J_{2} J_{1}

. Matching indices, not forward intuition, dictates the order.

"The Jacobian determinant for polar coordinates is

r^{2}

It is

r

, not

r^{2}

det [cos θ sin θ - r sin θ r cos θ] = r (cos^{2} θ + sin^{2} θ) = r

. That single

r

is the

r d r d θ

factor in polar integration.

Why questions

Why is the Jacobian "the" derivative for vector functions rather than just a bag of partials?

Because it is the unique linear map that best approximates

f

near a point:

f (x + h) \approx f (x) + J h

. That single defining property is what makes it play the role

f^{'} (x)

plays in one dimension. See Linear approximation and Taylor series.

Why does the chain rule become a matrix product of Jacobians?

Composing functions composes their best-linear approximations, and composing linear maps is matrix multiplication. So

J_{g \circ f} = J_{g} J_{f}

— see The chain rule (multivariable).

Why do we care about

∣ det J ∣

at all in integration?

When you change variables, a tiny input cell of volume

d V

maps to an output cell of volume

∣ det J ∣ d V

; that factor rescales the integral so it counts the same "stuff." This is Change of variables in integration.

Why is a Jacobian-vector product cheaper than forming the full Jacobian?

You often only need

J v

for a chosen direction

v

, and forward-mode autodiff computes that directly without ever materializing the

m \times n

table — see Jacobian-vector products (forward-mode autodiff).

Why does the gradient equal a Jacobian row and not a column?

The Jacobian must left-multiply

h

to give a change in

f

, so its single row dotted with

h

produces the directional change; the gradient is the transpose of that row, chosen as a column so

\nabla f \cdot h

reads naturally.

Why is the second-derivative analogue (the Hessian matrix) symmetric while a general Jacobian is not?

The Hessian is the Jacobian of a gradient, whose entries are

\partial^{2} f / \partial x_{i} \partial x_{j}

; mixed partials commute (Schwarz), forcing symmetry. A general Jacobian's entries

\partial f_{i} / \partial x_{j}

mix different outputs, so no such symmetry applies.

Edge cases

What is the Jacobian when

n = m = 1

1 \times 1

matrix holding the single number

f^{'} (x)

— the ordinary derivative recovered as the smallest possible Jacobian.

What does the Jacobian of a constant function

f (x) = c

look like?

The zero matrix of shape

m \times n

: no output responds to any input, so every partial is

0

What is the Jacobian of the identity map

f (x) = x

The

n \times n

identity matrix

I

, since

\partial x_{i} / \partial x_{j} = 1

when

i = j

and

0

otherwise — nudging an input moves exactly its own output.

Can a square Jacobian have

det J < 0

, and what does that mean?

Yes; the sign flips when the map reverses orientation (like a mirror). Volume still scales by

∣ det J ∣

, but a right-handed patch becomes left-handed.

For an affine map

f (x) = A x + b

, does

b

appear in

J

No. Constants have zero derivative, so

J = A

regardless of

b

; the offset shifts outputs but never changes local slopes.

At a point where

f

is not differentiable (a kink, like ReLU at

0

), does the Jacobian exist?

Strictly no — the linear approximation isn't unique there, so the true Jacobian is undefined. In practice frameworks pick a subgradient (e.g.

0

1

) as a convention, which powers Backpropagation through non-smooth layers.

Recall One-line self-test

Cover everything: state the shape, the $(i, j)$ entry, the linearization, the chain-rule order, and when $det J$ exists — in one breath. Answer ::: Shape $m \times n$ ; entry $J_{ij} = \partial f_{i} / \partial x_{j}$ ; $f (x + h) \approx f (x) + J h$ ; chain rule $J_{g \circ f} = J_{g} J_{f}$ (outer left); $det J$ only if $m = n$ .