Worked examples — The Jacobian matrix

The scenario matrix

Before working anything, let us list every case class a Jacobian problem can fall into. This is a checklist, not just decoration — read it like this:

• The # column gives each case a short letter name (A through I). We only use these letters as bookmarks so that each worked example can say "this one covers case A."
• The Case class column (in bold) is the plain-English name of the situation.
• The What makes it special column is the one idea to remember for that case.
• The Covered by column tells you exactly which worked example below drills that case, so you can jump straight to it.

Every cell A–I is hit at least once below.

Example 1 — square, generic point, with area scaling (Cell A)

Step 1 — Take all four partials. $\frac{\partial f_1}{\partial x}=2x,\frac{\partial f_1}{\partial y}=-2y,\frac{\partial f_2}{\partial x}=2y,\frac{\partial f_2}{\partial y}=2x.$ Why this step? By definition $J_{ij}=\partial f_i/\partial x_j$: differentiate output $i$ w.r.t. input $j$, holding the other input fixed.

Step 2 — Assemble the matrix. $J(x,y)=\left[\begin{array}{cc}2x& -2y\\ 2y& 2x\end{array}\right],J(2,1)=\left[\begin{array}{cc}4& -2\\ 2& 4\end{array}\right].$ Why this step? Rows = outputs, columns = inputs. Plugging in the point turns symbolic slopes into the concrete local linear map.

Step 3 — Area factor via the determinant. $\det J(2,1)=(4)(4)-(-2)(2)=16+4=20.$ Why this step? For a square Jacobian, $|\det J|$ is the local area-scaling factor — how much a tiny patch's area is multiplied.

The figure below shows this. On the left, a small coral (red) unit square sits at the input point $(2,1)$. On the right is its image after the linear map $J(2,1)$: a lavender (blue) parallelogram whose area is $20$ times bigger. Notice the parallelogram is both stretched and rotated — the two effects that $J$ packs into one matrix.

Example 2 — non-square TALL map, a surface in 3D (Cell B)

Step 1 — Shape first. Inputs $n=2$, outputs $m=3$, so $J$ is a $3\times 2$ matrix. Why this step? Rows are outputs (3), columns are inputs (2). $J$ must send $\Delta \mathbf{x}\in {\mathbb{R}}^2$ to $\Delta \mathbf{f}\in {\mathbb{R}}^3$: $(3\times 2)(2\times 1)=(3\times 1)$. ✔

Step 2 — Partials. $J(u,v)=\left[\begin{array}{cc}1& 0\\ 0& 1\\ 2u& 2v\end{array}\right],J(1,1)=\left[\begin{array}{cc}1& 0\\ 0& 1\\ 2& 2\end{array}\right].$ Why this step? Each column tells you the 3D velocity of the surface point as you push one input. Column 1 = $\partial \mathbf{f}/\partial u=(1,0,2u)$; column 2 = $\partial \mathbf{f}/\partial v=(0,1,2v)$. These two columns span the tangent plane to the surface.

Step 3 — Determinant? No. $\det$ needs a square matrix ($m=n$). Here $3\ne 2$, so there is no determinant. Why this step? Determinant measures how one $n$-volume maps to another $n$-volume; that only makes sense when input- and output-dimension match.

Example 3 — square but SINGULAR point (Cells A + D)

Step 1 — Plug in. $J(0,0)=\left[\begin{array}{cc}0& 0\\ 0& 0\end{array}\right],\det J(0,0)=0.$ Why this step? $\det J=4(x^2+y^2)$ from Ex 1, which is $0$ only at the origin.

Step 2 — Interpret $\det J=0$. The local linear map crushes every input direction to (nearly) zero — a tiny patch's first-order area collapses. Why this step? $|\det J|$ is the area factor; a factor of $0$ means the linear approximation flattens the neighbourhood. (The true map $z\mapsto z^2$ actually doubles angles and squishes near $0$ — the linear part just can't see that curvature.)

The figure below contrasts the two behaviours side by side. On the left, a small coral square sits right at the origin. On the right, its image under $z\mapsto z^2$ is squished almost to a point — unlike the fat parallelogram we saw in Example 1, here the area is driven toward zero because $\det J=0$.

Example 4 — non-square WIDE map with a nullspace (Cell C)

Step 1 — Shape. $m=1$, $n=3$ → $J$ is a $1\times 3$ matrix, a row vector. Why this step? Scalar output ⇒ single row. This is the gradient transposed: $J=(\nabla f{)}^{\top }$.

Step 2 — Entries. $J=\left[\begin{array}{ccc}1& 2& 3\end{array}\right].$ Why this step? $\partial f/\partial x=1,\partial f/\partial y=2,\partial f/\partial z=3$.

Step 3 — Nullspace = "silent" directions. $J\mathbf{h}=0$ means $h_1+2h_2+3h_3=0$. Any $\mathbf{h}$ on that plane changes nothing to first order. Why this step? $\Delta f\approx J\mathbf{h}$; if $J\mathbf{h}=0$, the output holds still. Example: $\mathbf{h}=(2,-1,0)$ gives $2-2+0=0$. ✔

Example 5 — zero / degenerate input where a partial misbehaves (Cell E)

Step 1 — Write the full Jacobian. $J(r,\theta )=\left[\begin{array}{cc}\frac{\partial (r\cos \theta )}{\partial r}& \frac{\partial (r\cos \theta )}{\partial \theta }\\ \frac{\partial (r\sin \theta )}{\partial r}& \frac{\partial (r\sin \theta )}{\partial \theta }\end{array}\right]=\left[\begin{array}{cc}\cos \theta & -r\sin \theta \\ \sin \theta & r\cos \theta \end{array}\right].$ Why this step? We need every entry visible before we can see which part collapses. Column 1 is $\partial \mathbf{f}/\partial r$ and column 2 is $\partial \mathbf{f}/\partial \theta$.

Step 2 — Determinant. $\det J=(\cos \theta )(r\cos \theta )-(-r\sin \theta )(\sin \theta )=r{\cos }^2\theta +r{\sin }^2\theta =r.$ At $r=0$: $\det J=0$. Why this step? At $r=0$ the area factor vanishes — a singular point.

Step 3 — Why it's degenerate. Set $r=0$ in the matrix: $J(0,\theta )=\left[\begin{array}{cc}\cos \theta & 0\\ \sin \theta & 0\end{array}\right].$ The entire second column is zero for every $\theta$. So the $\theta$-direction carries no information: at $r=0$ every angle $\theta$ maps to the same point $(0,0)$, and the map isn't locally one-to-one. Why this step? $\det J=0$ (equivalently, a zero column ⇒ rank $1$ not $2$) is the algebraic signal that the local linear map is not invertible — the whole $\theta$-column of information collapses.

Step 4 — Away from the origin it's fine. For any $r>0$, $\det J=r>0$: invertible, orientation-preserving. Why this step? Confirms the degeneracy is isolated to $r=0$, which is why polar integration writes the factor as $rdrd\theta$ — the $r$ vanishing at the origin is a genuine feature, not a bug.

Example 6 — elementwise / diagonal Jacobian: softmax's cheaper cousin (Cell F)

Step 1 — Off-diagonals vanish. $a_i=\tanh (z_i)$ depends only on $z_i$, so $\partial a_i/\partial z_j=0$ for $i\ne j$. Why this step? Elementwise activations have no cross-talk between coordinates → Jacobian is diagonal, which is why frameworks store it as a vector, not a full matrix.

Step 2 — Diagonal entries. $\frac{d}{dt}\tanh t=1-{\tanh }^{2}t$. So $J_{\sigma }=\mathrm{diag}(1-{\tanh }^{2}0,1-{\tanh }^{2}1).$

Step 3 — Numbers. $\tanh 0=0\Rightarrow 1-0=1$. $\tanh 1\approx 0.7616\Rightarrow 1-0.7616^2\approx 0.4200$. $J_{\sigma }\approx \left[\begin{array}{cc}1& 0\\ 0& 0.4200\end{array}\right].$ Why this step? Each diagonal entry is the slope of the activation at that coordinate — its local gain in backprop.

Example 7 — chain rule, order matters (Cell G)

Step 1 — Inner Jacobian. $\partial \mathbf{z}/\partial \mathbf{x}=W$ (since $z_i={\sum }_j{W}_{ij}{x}_j$). Why this step? Differentiating a linear map returns the matrix itself.

Step 2 — Evaluate $\mathbf{z}$. $\mathbf{x}=(0,0)\Rightarrow \mathbf{z}=W\mathbf{x}=(0,0)$. Why this step? We need $\mathbf{z}$ to evaluate the activation's slopes. Both entries are $0$, where ${\tanh }^{\prime }=1$.

Step 3 — Outer (diagonal) Jacobian. $\partial \mathbf{a}/\partial \mathbf{z}=\mathrm{diag}(1-{\tanh }^{2}0,1-{\tanh }^{2}0)=\mathrm{diag}(1,1)=I$.

Step 4 — Compose, outer on the LEFT (see The chain rule (multivariable)). $J=\underset{\text{outer}}{\underbrace{\mathrm{diag}({\sigma }^{\prime }(\mathbf{z}))}}\underset{\text{inner}}{\underbrace{W}}=IW=\left[\begin{array}{cc}1& 2\\ 0& 3\end{array}\right].$ Why this step? $\mathbf{h}=\mathbf{g}\circ \mathbf{f}\Rightarrow J_{\mathbf{h}}=J_{\mathbf{g}}{J}_{\mathbf{f}}$. The outer function's Jacobian multiplies on the left; the reversed product $W\mathrm{diag}$ would still be defined here but is wrong in general.

Example 8 — real-world word problem, with units (Cell H)

Step 1 — Jacobian symbolically. $J=\left[\begin{array}{cc}\partial f_1/\partial x_1& \partial f_1/\partial x_2\\ \partial f_2/\partial x_1& \partial f_2/\partial x_2\end{array}\right]=\left[\begin{array}{cc}3& 2\\ 0.5x_2& 0.5x_1\end{array}\right].$ Why this step? Column 1 = "response of every output to a nudge in machine time" — exactly what the question asks.

Step 2 — Evaluate at $(10,4)$. $J(10,4)=\left[\begin{array}{cc}3& 2\\ 2& 5\end{array}\right].$ Why this step? $0.5x_2=0.5(4)=2$; $0.5x_1=0.5(10)=5$.

Step 3 — Nudge machine time by $\Delta x_1=1$ hr, $\Delta x_2=0$. $\Delta \mathbf{f}\approx J\left[\begin{array}{c}1\\ 0\end{array}\right]=\left[\begin{array}{c}3\\ 2\end{array}\right].$ So $\approx +3$ widgets and $\approx +2$ kJ heat per extra hour. Why this step? Multiplying $J$ by the input-change vector $(1,0)$ picks out column 1 — the machine-time column.

Example 9 — exam twist: the transpose / commutativity trap (Cell I)

Step 1 — Fix the shape error. $g:{\mathbb{R}}^2\to \mathbb{R}$ has $m=1$, so its Jacobian is a $1\times 2$ row, not a column: $J_g=\left[\begin{array}{cc}2x_1& 2x_2\end{array}\right]=\left[\begin{array}{cc}2& 4\end{array}\right]\text{ at }(1,2).$ Why this step? Rows = outputs. One scalar output ⇒ one row. The column $(2x_1,2x_2{)}^{\top }$ the student wrote is the gradient; the Jacobian is its transpose.

Step 2 — Get the composition order right. The composite is $\mathbf{h}=\mathbf{f}\circ g$, meaning "apply $g$ first, then $\mathbf{f}$." The chain rule puts the outer function's Jacobian on the LEFT: $J_{\mathbf{h}}=J_{\mathbf{f}}(g(\mathbf{x}))J_g(\mathbf{x}).$ Check the shapes conform: $\mathbf{f}$ takes a scalar and returns 2 outputs, so $J_{\mathbf{f}}$ is $2\times 1$; $J_g$ is $1\times 2$. The product is $(2\times 1)(1\times 2)=(2\times 2)$. The reversed order $J_g{J}_{\mathbf{f}}$ would be $(1\times 2)(2\times 1)=(1\times 1)$ — the wrong shape, so order genuinely matters. Why this step? Matrix multiplication is not commutative; only the outer-on-the-left order has conforming inner dimensions.

Step 3 — Evaluate the inner scalar. $u=g(1,2)=1^2+2^2=5$. Why this step? We must feed $g$'s output value into $J_{\mathbf{f}}$ before differentiating the outer map.

Step 4 — Outer Jacobian at $u=5$. $\mathbf{f}(u)=(u^3,\sin u)$, so $J_{\mathbf{f}}(u)=\left[\begin{array}{c}3u^2\\ \cos u\end{array}\right]\Rightarrow J_{\mathbf{f}}(5)=\left[\begin{array}{c}75\\ \cos 5\end{array}\right].$ Why this step? $3\cdot 5^2=75$; the second output's slope is $\cos u$.

Step 5 — Multiply, outer on the left. $J_{\mathbf{h}}=J_{\mathbf{f}}(5)J_g(1,2)=\left[\begin{array}{c}75\\ \cos 5\end{array}\right]\left[\begin{array}{cc}2& 4\end{array}\right]=\left[\begin{array}{cc}150& 300\\ 2\cos 5& 4\cos 5\end{array}\right].$ Why this step? An $(2\times 1)$ column times a $(1\times 2)$ row is an outer product, giving the full $2\times 2$ Jacobian of the composite.

Active-recall