The Jacobian matrix: Exercises — AI-ML deep dive

Quick reminder of the one rule everything rests on. For a function that eats a vector and returns a vector, $f : R^{n} \to R^{m}$ , the Jacobian is the table of partial derivatives

J_{ij} = \frac{\partial f _{i}}{\partial x _{j}}, J \in R^{m \times n},

with ==rows = outputs, columns = inputs==. It is the best linear approximation: $f (x + h) \approx f (x) + J h$ .

Level 1 — Recognition

Exercise 1.1 — Read off the shape

Let $f : R^{4} \to R^{2}$ . Without computing anything, state the shape of its Jacobian and say what a single row and a single column represent.

Recall Solution

Shape. Rows = number of outputs $= 2$ ; columns = number of inputs $= 4$ . So $J \in R^{2 \times 4}$ . What we did & why: the shape is fixed before any calculus, purely by counting outputs and inputs. This is the "ROC — Rows are Outputs, Columns are inputs" mnemonic.

A row ( $1 \times 4$ ) is $(\nabla f_{i})^{⊤}$ : the gradient of one output $f_{i}$ , telling how that single output responds to all $4$ inputs.
A column ( $2 \times 1$ ) is $\partial f / \partial x_{j}$ : how both outputs respond to nudging one input $x_{j}$ . Check: $J h$ must send an input-direction $h \in R^{4}$ to an output-change in $R^{2}$ . $(2 \times 4) (4 \times 1) = (2 \times 1)$ . ✔

Exercise 1.2 — Which entry is which?

For $f (x, y, z) = [x y y + z]$ , which partial derivative lives at position $J_{2, 3}$ (row 2, column 3)?

Recall Solution

Position $(i, j) = (2, 3)$ means output $f_{2}$ , input $x_{3}$ . Here $f_{2} = y + z$ and the third input is $z$ . $J_{2, 3} = \frac{\partial f _{2}}{\partial z} = \frac{\partial ( y + z )}{\partial z} = 1.$ Why: the first index always picks the output (which row), the second index picks the input (which column). Getting this ordering automatic is the whole point of Level 1.

Level 2 — Application

Exercise 2.1 — Compute a $2 \to 2$ Jacobian

Let $f (x, y) = [x^{2} + y e^{x} y]$ . Find $J (x, y)$ and evaluate at $(0, 3)$ .

Recall Solution

Differentiate each output with respect to each input, holding the other input fixed:

$\partial f_{1} / \partial x = 2 x$ , $\partial f_{1} / \partial y = 1$
$\partial f_{2} / \partial x = e^{x} y$ , $\partial f_{2} / \partial y = e^{x}$

$J (x, y) = [2 x e^{x} y 1 e^{x}], J (0, 3) = [0311] .$ Why each step: every entry is "pick an output, differentiate w.r.t. one input, treat the other input as a constant." Substituting $(0, 3)$ (using $e^{0} = 1$ ) turns symbolic slopes into the concrete local linear map.

Exercise 2.2 — Linear approximation with the Jacobian

Using $J (0, 3)$ from Exercise 2.1 and $f (0, 3) = [33]$ , estimate $f (0.1, 3.2)$ .

Recall Solution

The step from the base point is $h = [0.1 0.2]$ . The linearization says $f (x + h) \approx f (x) + J h$ :

J (0, 3) h = [0311] [0.1 0.2] = [0 (0.1) + 1 (0.2) 3 (0.1) + 1 (0.2)] = [0.2 0.5] .

f (0.1, 3.2) \approx [33] + [0.2 0.5] = [3.2 3.5] .

Why the Jacobian and not a plain slope? With two outputs there is no single "slope" number — the Jacobian is the matrix that plays the role of $f^{'} (x)$ , so multiplying it by the nudge $h$ predicts the change in both outputs at once. (True value $\approx (3.21, 3.536)$ : the linear estimate is close because $h$ is small — see Linear approximation and Taylor series.)

Level 3 — Analysis

Exercise 3.1 — A non-square Jacobian and why it has no determinant

Let $f (x, y, z) = [x + 2 y y z]$ . Find $J$ , state its shape, and explain in one sentence why $det J$ is meaningless here.

Recall Solution

$J (x, y, z) = \frac{\partial f _{1}}{\partial x} \frac{\partial f _{2}}{\partial x} \frac{\partial f _{1}}{\partial y} \frac{\partial f _{2}}{\partial y} \frac{\partial f _{1}}{\partial z} \frac{\partial f _{2}}{\partial z} = [10 2 z 0 y] .$ Shape: $2$ outputs, $3$ inputs $\Rightarrow J \in R^{2 \times 3}$ . Why no determinant: a determinant is only defined for a square matrix. Here $m = 2 \neq = n = 3$ , so there is no $det J$ . (Determinants measure how a map scales volume from a space back into itself; a map $R^{3} \to R^{2}$ changes the number of dimensions, so "volume scaling" has no single meaning — see Change of variables in integration.)

Exercise 3.2 — Diagonal Jacobian of an elementwise map

Let $a = σ (z)$ apply the same scalar function $σ$ to each entry of $z \in R^{3}$ , i.e. $a_{i} = σ (z_{i})$ . Show the Jacobian $\partial a / \partial z$ is diagonal, and write it for $σ (t) = tanh t$ .

Recall Solution

Entry $J_{ij} = \partial a_{i} / \partial z_{j}$ . Since $a_{i} = σ (z_{i})$ depends only on $z_{i}$ :

If $i \neq = j$ : $a_{i}$ doesn't contain $z_{j}$ , so $J_{ij} = 0$ (off-diagonal vanishes).
If $i = j$ : $J_{ii} = σ^{'} (z_{i})$ .

Hence $J = diag (σ^{'} (z_{1}), σ^{'} (z_{2}), σ^{'} (z_{3}))$ . For $σ = tanh$ , recall $tanh^{'} (t) = 1 - tanh^{2} (t)$ , so $J = diag (1 - tanh^{2} z_{1}, 1 - tanh^{2} z_{2}, 1 - tanh^{2} z_{3}) .$ Why this matters: elementwise activations in a neural net always give diagonal Jacobians — that is exactly why Backpropagation can multiply by them cheaply (a diagonal times a vector is just entrywise scaling).

Level 4 — Synthesis

Exercise 4.1 — Chain rule as a matrix product

Let $g (u, v) = [u + v uv]$ and $f (x) = [x^{2} x]$ , so $h = g \circ f : R \to R^{2}$ . Compute $J_{h}$ at $x = 2$ using the Jacobian chain rule $J_{h} = J_{g} J_{f}$ , then verify by substituting first.

Recall Solution

Inner Jacobian. $f : R \to R^{2}$ , so $J_{f}$ is $2 \times 1$ : $J_{f} (x) = [2 x 1], J_{f} (2) = [41] .$ Outer Jacobian in the variables $(u, v)$ : $g : R^{2} \to R^{2}$ , so $2 \times 2$ : $J_{g} (u, v) = [1 v 1 u] .$ At $x = 2$ : $u = f_{1} = x^{2} = 4$ , $v = f_{2} = x = 2$ , so $J_{g} = [1214]$ . Multiply (outer on the LEFT): $J_{h} (2) = J_{g} J_{f} = [1214] [41] = [1 (4) + 1 (1) 2 (4) + 4 (1)] = [512] .$ Verify by substituting first. $h (x) = g (x^{2}, x) = [x^{2} + x x^{2} \cdot x] = [x^{2} + x x^{3}]$ , so $J_{h} (x) = [2 x + 1 3 x^{2}], J_{h} (2) = [512] . ✔$ Why this works: the multivariable The chain rule (multivariable) says composing functions composes their linear approximations, and composing linear maps is matrix multiplication. Shapes only fit as $(2 \times 2) (2 \times 1) = (2 \times 1)$ , forcing the outer Jacobian on the left.

Exercise 4.2 — Jacobian of a linear layer + activation

Let $z = W x$ with $W = [10 2 - 1]$ and then $a = σ (z)$ elementwise with $σ (t) = tanh t$ . Give the full Jacobian $\partial a / \partial x$ symbolically, then evaluate at $x = [10]$ .

Recall Solution

Two building blocks:

$\partial z / \partial x = W$ (because $z_{i} = \sum_{j} W_{ij} x_{j} \Rightarrow \partial z_{i} / \partial x_{j} = W_{ij}$ ).
$\partial a / \partial z = diag (σ^{'} (z_{1}), σ^{'} (z_{2})) = diag (1 - tanh^{2} z_{1}, 1 - tanh^{2} z_{2})$ (Exercise 3.2).

Chain rule (activation is the outer map, so it goes on the left): $\frac{\partial a}{\partial x} = diag (σ^{'} (z)) W .$ Evaluate at $x = (1, 0)$ : $z = W x = [1 \cdot 1 + 2 \cdot 0 0 \cdot 1 + (- 1) \cdot 0] = [10]$ . Then $σ^{'} (z_{1}) = 1 - tanh^{2} (1) \approx 1 - 0.761 6^{2} \approx 0.4200$ , and $σ^{'} (z_{2}) = 1 - tanh^{2} (0) = 1$ . $\frac{\partial a}{\partial x} \approx [0.4200 0 01] [10 2 - 1] = [0.4200 0 0.8400 - 1] .$ Why this is backprop: this product-of-Jacobians is precisely how gradients flow through one layer in Backpropagation and Jacobian-vector products (forward-mode autodiff).

Level 5 — Mastery

Exercise 5.1 — Determinant as local area scaling (polar map)

For the polar-to-Cartesian map $f (r, θ) = (r cos θ, r sin θ)$ , we have $det J = r$ . Interpret geometrically: how much does a tiny rectangle $d r \times d θ$ located at $(r, θ)$ get stretched in area, and what happens at $r = 0$ ?

Recall Solution

The Jacobian is $J = [cos θ sin θ - r sin θ r cos θ]$ , and $det J = cos θ \cdot r cos θ - (- r sin θ) \cdot sin θ = r cos^{2} θ + r sin^{2} θ = r .$ Geometry (see figure): a small patch that is $d r$ wide (radial) and $d θ$ tall (angular) maps to a curved patch in the plane. Its two edges have physical lengths $d r$ (radial) and $r d θ$ (arc length — the arc grows with radius). Area $\approx d r \cdot r d θ = r d r d θ$ , matching $∣ det J ∣ = r$ . Why the determinant answers this: for a square Jacobian, $∣ det J ∣$ is the factor by which any tiny volume is multiplied by the map. That is literally the $r d r d θ$ in polar integration — see Change of variables in integration. At $r = 0$ : $det J = 0$ . The map collapses the whole line $r = 0$ (every $θ$ ) onto the single origin point, so any tiny patch there is squashed to zero area. A zero determinant flags a non-invertible (degenerate) point — the polar coordinate breaks down at the origin because $θ$ is undefined there.

Exercise 5.2 — When the linear approximation is exact

Prove that for an affine map $f (x) = A x + b$ (with constant matrix $A \in R^{m \times n}$ and constant vector $b$ ), the Jacobian is $A$ everywhere, and the "approximation" $f (x + h) = f (x) + J h$ holds with exact equality.

Recall Solution

Component $i$ : $f_{i} (x) = \sum_{j} A_{ij} x_{j} + b_{i}$ . Then $\partial f_{i} / \partial x_{j} = A_{ij}$ (the $b_{i}$ is constant, drops out). So $J = A$ , constant for all $x$ . Now compute the exact difference: $f (x + h) = A (x + h) + b = A x + b + A h = f (x) + A h = f (x) + J h .$ No " $\approx$ " needed — it is " $=$ ". Why: the linearization keeps only first-order (straight-line) behavior. An affine map is already straight, so there is no curvature (second-order term) to discard. The Taylor series terminates; the Jacobian captures the map perfectly. This is the base case behind Linear approximation and Taylor series.

Exercise 5.3 — Jacobian-vector product without forming $J$

For $f (x, y) = [x^{2} y sin (x) + y]$ at the point $(x, y) = (\frac{π}{2}, 2)$ , compute the Jacobian-vector product $J v$ for the direction $v = [1 - 1]$ — the directional rate of change of both outputs.

Recall Solution

$J (x, y) = [2 x y cos x x^{2} 1] .$ At $(\frac{π}{2}, 2)$ : $2 x y = 2 \cdot \frac{π}{2} \cdot 2 = 2 π$ ; $x^{2} = \frac{π ^{2}}{4}$ ; $cos (\frac{π}{2}) = 0$ ; the last entry is $1$ .

J\mathbf{v}=\begin{bmatrix}2\pi & \tfrac{\pi^2}{4}\\ 0 & 1\end{bmatrix}\begin{bmatrix}1\\-1\end{bmatrix} =\begin{bmatrix}2\pi-\tfrac{\pi^2}{4}\\ -1\end{bmatrix}\approx\begin{bmatrix}3.815\\ -1\end{bmatrix}.$$ **Why "without forming $J$" matters:** in [[Jacobian-vector products (forward-mode autodiff)]] you compute $J\mathbf{v}$ directly by pushing the direction $\mathbf{v}$ through the function — you never build the full (possibly huge) matrix. Here we formed $J$ to check the arithmetic, but the *value* $J\mathbf{v}$ is what forward-mode autodiff returns cheaply. The first output rises at rate $\approx 3.815$, the second falls at rate $1$, when we move in direction $(1,-1)$.

Recall One-line self-quiz

Shape of $J$ for $f : R^{n} \to R^{m}$ ? ::: $m \times n$ (rows = outputs, columns = inputs). Chain rule for $h = g \circ f$ ? ::: $J_{h} = J_{g} J_{f}$ , outer on the left. When does $det J$ exist and what does $∣ det J ∣$ mean? ::: only if $J$ is square ( $m = n$ ); it is the local volume-scaling factor. When is $f (x + h) = f (x) + J h$ exact? ::: when $f$ is affine ( $A x + b$ ), because there is no curvature to drop.