Intuition The big picture (WHY)A single-input, single-output function f ( x ) f(x) f ( x ) f ′ ( x ) f'(x) f ′ ( x ) vector in and spit a vector out : f : R n → R m \mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m f : R n → R m table of partial derivatives , one for every (output, input) pair. That table is the Jacobian matrix . It's the best linear approximation of a vector-valued function near a point — the multi-dimensional generalization of f ′ ( x ) f'(x) f ′ ( x )
Definition Jacobian matrixFor f : R n → R m \mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m f : R n → R m f ( x ) = [ f 1 ( x 1 , … , x n ) ⋮ f m ( x 1 , … , x n ) ] , \mathbf{f}(\mathbf{x}) = \begin{bmatrix} f_1(x_1,\dots,x_n) \\ \vdots \\ f_m(x_1,\dots,x_n) \end{bmatrix}, f ( x ) = f 1 ( x 1 , … , x n ) ⋮ f m ( x 1 , … , x n ) , Jacobian J J J J f \mathbf{J_f} J f ∂ f ∂ x \dfrac{\partial \mathbf{f}}{\partial \mathbf{x}} ∂ x ∂ f m × n m \times n m × n J i j = ∂ f i ∂ x j . J_{ij} = \frac{\partial f_i}{\partial x_j}. J ij = ∂ x j ∂ f i . i i i f i f_i f i j j j x j x_j x j
J = [ ∂ f 1 ∂ x 1 ⋯ ∂ f 1 ∂ x n ⋮ ⋱ ⋮ ∂ f m ∂ x 1 ⋯ ∂ f m ∂ x n ] J = \begin{bmatrix}
\dfrac{\partial f_1}{\partial x_1} & \cdots & \dfrac{\partial f_1}{\partial x_n} \\[6pt]
\vdots & \ddots & \vdots \\[6pt]
\dfrac{\partial f_m}{\partial x_1} & \cdots & \dfrac{\partial f_m}{\partial x_n}
\end{bmatrix} J = ∂ x 1 ∂ f 1 ⋮ ∂ x 1 ∂ f m ⋯ ⋱ ⋯ ∂ x n ∂ f 1 ⋮ ∂ x n ∂ f m
Intuition Shape mnemonic — "rows are outputs, columns are inputs"The Jacobian has ==m m m (one per output) and n n n J ∈ R m × n J \in \mathbb{R}^{m\times n} J ∈ R m × n Δ x ∈ R n \Delta\mathbf{x}\in\mathbb{R}^n Δ x ∈ R n Δ f ∈ R m \Delta\mathbf{f}\in\mathbb{R}^m Δ f ∈ R m ( m × n ) ( n × 1 ) = ( m × 1 ) (m\times n)(n\times 1) = (m\times 1) ( m × n ) ( n × 1 ) = ( m × 1 )
Intuition Derivation ideaThe derivative of f f f x x x the number that makes the linear approximation work:
f ( x + h ) ≈ f ( x ) + f ′ ( x ) h . f(x+h)\approx f(x)+f'(x)\,h. f ( x + h ) ≈ f ( x ) + f ′ ( x ) h . J J J f ( x + h ) ≈ f ( x ) + J h . \mathbf{f}(\mathbf{x}+\mathbf{h}) \approx \mathbf{f}(\mathbf{x}) + J\,\mathbf{h}. f ( x + h ) ≈ f ( x ) + J h .
Step 1 — Look at one output at a time. Take component f i f_i f i
f i ( x + h ) ≈ f i ( x ) + ∑ j = 1 n ∂ f i ∂ x j h j . f_i(\mathbf{x}+\mathbf{h}) \approx f_i(\mathbf{x}) + \sum_{j=1}^{n}\frac{\partial f_i}{\partial x_j}\,h_j. f i ( x + h ) ≈ f i ( x ) + j = 1 ∑ n ∂ x j ∂ f i h j . Why this step? This is just the multivariable Taylor / total-differential rule: perturbing each input x j x_j x j h j h_j h j f i f_i f i ∂ f i / ∂ x j \partial f_i/\partial x_j ∂ f i / ∂ x j
Step 2 — Recognize the sum as a dot product.
∑ j ∂ f i ∂ x j h j = [ ∂ f i ∂ x 1 ⋯ ∂ f i ∂ x n ] ⏟ row i h = ( ∇ f i ) ⊤ h . \sum_{j}\frac{\partial f_i}{\partial x_j}h_j = \underbrace{\Big[\tfrac{\partial f_i}{\partial x_1}\ \cdots\ \tfrac{\partial f_i}{\partial x_n}\Big]}_{\text{row }i}\,\mathbf{h} = (\nabla f_i)^\top \mathbf{h}. j ∑ ∂ x j ∂ f i h j = row i [ ∂ x 1 ∂ f i ⋯ ∂ x n ∂ f i ] h = ( ∇ f i ) ⊤ h . Why this step? A weighted sum of h j h_j h j h \mathbf{h} h f i f_i f i
Step 3 — Stack all m m m Doing this for every output simultaneously:
f ( x + h ) ≈ f ( x ) + J h , J = [ ( ∇ f 1 ) ⊤ ⋮ ( ∇ f m ) ⊤ ] . \mathbf{f}(\mathbf{x}+\mathbf{h}) \approx \mathbf{f}(\mathbf{x}) + J\mathbf{h}, \qquad
J = \begin{bmatrix}(\nabla f_1)^\top\\ \vdots \\ (\nabla f_m)^\top\end{bmatrix}. f ( x + h ) ≈ f ( x ) + J h , J = ( ∇ f 1 ) ⊤ ⋮ ( ∇ f m ) ⊤ . Why this step? Stacking the row-approximations is the same as one matrix equation. The matrix whose rows are the gradients is the Jacobian. ∎
Worked example Example 1 — a2 → 2 2\to 2 2 → 2 f ( x , y ) = [ x 2 y x + sin y ] \mathbf{f}(x,y)=\begin{bmatrix} x^2 y \\ x+\sin y\end{bmatrix} f ( x , y ) = [ x 2 y x + sin y ] J J J ( 1 , 0 ) (1,0) ( 1 , 0 )
Partials:
∂ f 1 / ∂ x = 2 x y \partial f_1/\partial x = 2xy ∂ f 1 / ∂ x = 2 x y ∂ f 1 / ∂ y = x 2 \quad \partial f_1/\partial y = x^2 ∂ f 1 / ∂ y = x 2 ∂ f 2 / ∂ x = 1 \partial f_2/\partial x = 1 ∂ f 2 / ∂ x = 1 ∂ f 2 / ∂ y = cos y \quad \partial f_2/\partial y = \cos y ∂ f 2 / ∂ y = cos y
J ( x , y ) = [ 2 x y x 2 1 cos y ] , J ( 1 , 0 ) = [ 0 1 1 1 ] . J(x,y)=\begin{bmatrix} 2xy & x^2 \\ 1 & \cos y\end{bmatrix}, \qquad J(1,0)=\begin{bmatrix} 0 & 1 \\ 1 & 1\end{bmatrix}. J ( x , y ) = [ 2 x y 1 x 2 cos y ] , J ( 1 , 0 ) = [ 0 1 1 1 ] . Why each step? Each entry is "output i i i j j j ( 1 , 0 ) (1,0) ( 1 , 0 )
Worked example Example 2 — chain rule via Jacobians (matches backprop)Let z = W x \mathbf{z}=W\mathbf{x} z = W x W ∈ R m × n W\in\mathbb{R}^{m\times n} W ∈ R m × n a = σ ( z ) \mathbf{a}=\sigma(\mathbf{z}) a = σ ( z )
J z = ∂ z ∂ x = W J_{\mathbf{z}}=\dfrac{\partial \mathbf{z}}{\partial \mathbf{x}}=W J z = ∂ x ∂ z = W Why? z i = ∑ j W i j x j z_i=\sum_j W_{ij}x_j z i = ∑ j W ij x j ∂ z i / ∂ x j = W i j \partial z_i/\partial x_j = W_{ij} ∂ z i / ∂ x j = W ij J σ = ∂ a ∂ z = diag ( σ ′ ( z 1 ) , … , σ ′ ( z m ) ) J_{\sigma}=\dfrac{\partial \mathbf{a}}{\partial \mathbf{z}}=\operatorname{diag}\big(\sigma'(z_1),\dots,\sigma'(z_m)\big) J σ = ∂ z ∂ a = diag ( σ ′ ( z 1 ) , … , σ ′ ( z m ) ) Why? a i = σ ( z i ) a_i=\sigma(z_i) a i = σ ( z i ) z i z_i z i Full map x ↦ a \mathbf{x}\mapsto\mathbf{a} x ↦ a J = J σ J z = diag ( σ ′ ( z ) ) W . J = J_\sigma J_{\mathbf{z}} = \operatorname{diag}(\sigma'(\mathbf{z}))\,W. J = J σ J z = diag ( σ ′ ( z )) W .
Why this matters: this product-of-Jacobians is forward-mode differentiation of a neural net.
Worked example Example 3 — polar → Cartesian & the determinantf ( r , θ ) = ( r cos θ , r sin θ ) \mathbf{f}(r,\theta)=(r\cos\theta,\ r\sin\theta) f ( r , θ ) = ( r cos θ , r sin θ ) J = [ cos θ − r sin θ sin θ r cos θ ] , det J = r cos 2 θ + r sin 2 θ = r . J=\begin{bmatrix}\cos\theta & -r\sin\theta\\ \sin\theta & r\cos\theta\end{bmatrix},\quad \det J = r\cos^2\theta + r\sin^2\theta = r. J = [ cos θ sin θ − r sin θ r cos θ ] , det J = r cos 2 θ + r sin 2 θ = r . Why the determinant? For a square Jacobian, ∣ det J ∣ |\det J| ∣ det J ∣ local area/volume scaling factor — how much a tiny patch expands under f \mathbf f f r d r d θ r\,dr\,d\theta r d r d θ
Common mistake "Jacobian and gradient are the same thing."Why it feels right: for a scalar function they carry identical numbers. The fix: the gradient ∇ f \nabla f ∇ f column vector; for m = 1 m=1 m = 1 transpose (a row). More importantly, "gradient" only makes sense when output is scalar (m = 1 m=1 m = 1 m > 1 m>1 m > 1
Common mistake "The Jacobian isn × m n\times m n × m Why it feels right: people write inputs first out of habit. The fix: it's m × n m\times n m × n rows = outputs, columns = inputs — because it must map R n → R m \mathbb R^n\to\mathbb R^m R n → R m J h J\mathbf h J h h ∈ R n \mathbf h\in\mathbb R^n h ∈ R n
Common mistake "Chain rule for Jacobians isJ f J g J_f J_g J f J g Why it feels right: scalar chain rule d h d x = d h d g d g d x \frac{dh}{dx}=\frac{dh}{dg}\frac{dg}{dx} d x d h = d g d h d x d g The fix: matrix multiplication is NOT commutative. For h = g ∘ f \mathbf h=\mathbf g\circ\mathbf f h = g ∘ f outer function's Jacobian on the left : J h = J g J f J_{\mathbf h}=J_{\mathbf g}\,J_{\mathbf f} J h = J g J f
det J \det J det J Why it feels right: we take determinants all the time. The fix: determinant needs a square matrix, i.e. m = n m=n m = n J J J
Recall Feynman: explain it to a 12-year-oldImagine a machine that takes a few dials as input and lights up a few bulbs as output. If you nudge one dial a tiny bit, each bulb gets a little brighter or dimmer. The Jacobian is just the table of all those "how much brighter per nudge" numbers — one number for each (bulb, dial) pair. If you know this table, you can predict what happens for any small twist of the dials without running the machine: just multiply your twist by the table.
"ROC — Rows are Outputs, Columns are inputs." And: gradient stacks into Jacobian — stack the gradient-rows of each output.
What is the shape of the Jacobian of f : R n → R m \mathbf f:\mathbb R^n\to\mathbb R^m f : R n → R m m × n m\times n m × n — rows = outputs, columns = inputs.
Define the ( i , j ) (i,j) ( i , j ) J i j = ∂ f i / ∂ x j J_{ij}=\partial f_i/\partial x_j J ij = ∂ f i / ∂ x j .
What is row i i i The transpose of the gradient of output
f i f_i f i , i.e.
( ∇ f i ) ⊤ (\nabla f_i)^\top ( ∇ f i ) ⊤ .
For a scalar function (m = 1 m=1 m = 1 The Jacobian is a row vector equal to
( ∇ f ) ⊤ (\nabla f)^\top ( ∇ f ) ⊤ ; the gradient is its transpose (a column).
State the first-order linear approximation using the Jacobian. f ( x + h ) ≈ f ( x ) + J h \mathbf f(\mathbf x+\mathbf h)\approx \mathbf f(\mathbf x)+J\mathbf h f ( x + h ) ≈ f ( x ) + J h .
Chain rule for h = g ∘ f \mathbf h=\mathbf g\circ\mathbf f h = g ∘ f J h = J g J f J_{\mathbf h}=J_{\mathbf g}\,J_{\mathbf f} J h = J g J f (outer on the left; order matters).
What does ∣ det J ∣ |\det J| ∣ det J ∣ Local volume-scaling factor of the map; defined only when
J J J is square (
m = n m=n m = n ).
Jacobian of z = W x \mathbf z=W\mathbf x z = W x x \mathbf x x Jacobian of elementwise a = σ ( z ) \mathbf a=\sigma(\mathbf z) a = σ ( z ) z \mathbf z z diag ( σ ′ ( z 1 ) , … , σ ′ ( z m ) ) \operatorname{diag}(\sigma'(z_1),\dots,\sigma'(z_m)) diag ( σ ′ ( z 1 ) , … , σ ′ ( z m )) .
det J \det J det J ( r cos θ , r sin θ ) (r\cos\theta,r\sin\theta) ( r cos θ , r sin θ ) Scalar derivative f prime x
Vector function f R^n to R^m
Partial derivatives dfi dxj
Column j is input xj response
Linear approximation requirement
Per-output first order expansion
Row times h equals grad fi dot h
ML backprop and optimization
Intuition Hinglish mein samjhoDekho, ek normal function f ( x ) f(x) f ( x ) vector in, vector out hote hain: n n n m m m table banti hai, wahi hai Jacobian matrix , size m × n m \times n m × n
Iska asli funda ye hai: Jacobian function ka best linear approximation hai kisi point ke aas-paas. Matlab f ( x + h ) ≈ f ( x ) + J h \mathbf{f}(\mathbf{x}+\mathbf{h}) \approx \mathbf{f}(\mathbf{x}) + J\mathbf{h} f ( x + h ) ≈ f ( x ) + J h f ( x + h ) ≈ f ( x ) + f ′ ( x ) h f(x+h)\approx f(x)+f'(x)h f ( x + h ) ≈ f ( x ) + f ′ ( x ) h f ′ f' f ′ J J J
ML me iski importance? Backpropagation basically Jacobians ka multiplication hi hai. Linear layer z = W x \mathbf{z}=W\mathbf{x} z = W x W W W σ \sigma σ diagonal matrix diag ( σ ′ ( z i ) ) \operatorname{diag}(\sigma'(z_i)) diag ( σ ′ ( z i )) left me aata hai, J g J f J_{\mathbf g} J_{\mathbf f} J g J f
Do galtiyan jo sab karte hain: (1) Jacobian ko gradient samajh lena — gradient toh sirf scalar output (m = 1 m=1 m = 1 det J \det J det J square ho (m = n m=n m = n ∣ det J ∣ |\det J| ∣ det J ∣