Chain rule for multivariate functions (backprop foundation): Worked examples

Before anything, one plain-words reminder so no symbol is used unearned:

Definition The three symbols we will lean on

$\frac{\partial z}{\partial x}$ (a "partial derivative") = "if I nudge only $x$ by a tiny bit and hold every independent input frozen, how fast does $z$ move?" The curly $\partial$ instead of $d$ is the flag that says "other inputs exist, I'm holding them still." When there is only one input, no freezing is needed and we write the straight $d$ instead: $\frac{d z}{d x}$ . So $d$ = single-variable chain, $\partial$ = multivariate chain — you'll see the switch happen in Example 1 (one input, $d$ ) versus Example 2 onward (many inputs, $\partial$ ).
A path = one chain of arrows in the diagram of who-depends-on-whom, e.g. $x \to u \to z$ .
$σ (a) = \frac{1}{1 + e ^{- a}}$ = the sigmoid, an S-shaped squashing function (see Activation Functions). Its derivative is $σ^{'} (a) = σ (a) (1 - σ (a))$ ; we prove this by hand in Example 5 (Step 1b) and it is machine-checked (see box below).

The Cell column names the class of scenario each example belongs to — the letters A–H are just labels for the eight rows of the matrix below, so that every worked example can announce "this one hits Cell C" and you can see the whole menu is covered with none skipped.

The scenario matrix

Every problem this topic can throw is one (or a mix) of the cells below. "Cell" here means one row of this table — a single named class of situation, labelled A through H. Each worked example names the cell(s) it lands in.

Cell (scenario class)	Situation	What's tricky	Example
A	Single path $x \to u \to z$	just multiply — no sum	Ex 1
B	Two paths $x \to u \to z$ and $x \to v \to z$	must add both	Ex 2
C	Paths cancel (contributions opposite sign)	net derivative can be $0$	Ex 3
D	Degenerate / zero link ( $\partial u / \partial x = 0$ )	a whole path vanishes	Ex 4
E	Saturated nonlinearity ( $h (1 - h) \to 0$ )	gradient dies — vanishing gradient	Ex 5
F	Real-world word problem (rates over time)	build the graph from a story	Ex 6
G	Vector/Jacobian, exam twist	order of matrix multiply matters	Ex 7
H	Limiting behaviour (large-input, saturation limit)	reason about $\to 0$ or $\to \infty$	Ex 8

Example 1 — Cell A: the single path

Worked example Single-path chain rule

Given: $z = ln (u)$ where $u = 3 x^{2} + 1$ . Find $\frac{d z}{d x}$ at $x = 1$ .

Forecast (guess first): Only one road from $x$ to $z$ , so there is nothing to add. Do you expect the answer to be positive, negative, or zero at $x = 1$ ?

Note on notation: $x$ is the only input here, so we use the straight $d$ (single-variable). From Example 2 on there will be more than one input, and we switch to the curly $\partial$ .

Step 1. $\frac{d z}{d u} = \frac{1}{u}$ . Why this step? This is the outer link $u \to z$ ; the derivative of $ln$ is $1/ u$ .

Step 2. $\frac{d u}{d x} = 6 x$ . Why this step? This is the inner link $x \to u$ ; differentiate $3 x^{2} + 1$ .

Step 3. Multiply along the single path: $\frac{d z}{d x} = \frac{1}{u} \cdot 6 x = \frac{6 x}{3 x ^{2} + 1} .$ Why this step? One path ⇒ pure product, no summing.

Step 4. At $x = 1$ : $u = 4$ , so $\frac{d z}{d x} = \frac{6}{4} = 1.5$ .

Verify: Sanity-check the sign — at $x = 1$ , $x$ increasing makes $u$ increase makes $ln u$ increase, so the derivative should be positive. It is $+ 1.5$ . ✓

Example 2 — Cell B: two paths, both add

Worked example Two paths that reinforce

Given: $z = u v$ where $u = x + y$ and $v = x - y$ . Find $\frac{\partial z}{\partial x}$ at $(x, y) = (3, 1)$ .

Note on notation: now there are two inputs $x$ and $y$ , so $z$ has more than one direction to move in — we switch to the curly $\partial$ to signal "hold the other input still."

Forecast: $x$ reaches $z$ through two intermediates. You must add two products. Will they help or hurt each other here?

Step 1. Path derivatives out of $z$ : $\frac{\partial z}{\partial u} = v$ , $\frac{\partial z}{\partial v} = u$ . Why this step? $z = uv$ , differentiate treating the other factor as a constant.

Step 2. Path derivatives into the intermediates: $\frac{\partial u}{\partial x} = 1$ , $\frac{\partial v}{\partial x} = 1$ . Why this step? These are the two roads $x \to u$ and $x \to v$ .

Step 3. Sum over paths: $\frac{\partial z}{\partial x} = v \cdot 1 + u \cdot 1 = u + v = (x + y) + (x - y) = 2 x .$ Why this step? Two paths ⇒ add. Forgetting one term is the #1 mistake.

Step 4. At $(3, 1)$ : $2 x = 6$ .

Verify: Substitute first: $z = (x + y) (x - y) = x^{2} - y^{2}$ , so $\partial z / \partial x = 2 x = 6$ . ✓ Both paths pointed the same way and reinforced.

Read the figure below before moving on. It draws the two roads explicitly: $x$ (white) branches up to $u$ (blue) and down to $v$ (green), and both roads rejoin at $z$ (yellow). The blue and green edge labels are the link derivatives from Steps 1–2; the total $\partial z / \partial x$ at the bottom is what you get by walking each coloured road (multiplying its two labels) and then adding the two roads. Trace the blue road with your finger, then the green — the addition of exactly these two walks is the whole content of Cell B.

Figure — Chain rule for multivariate functions (backprop foundation)

Figure 1 — Two paths from input $x$ to output $z$ : multiply along each coloured road, add across roads.

Example 3 — Cell C: paths that cancel to zero

Worked example When contributions fight and net to zero

Given: $z = u^{2} - v^{2}$ with $u = x + y$ , $v = x - y$ . Find $\frac{\partial z}{\partial x}$ at $(x, y) = (0, 5)$ .

Forecast: Two paths again — but now with a minus in $z$ . Do you think the total can be zero even though each path alone is nonzero?

Step 1. $\frac{\partial z}{\partial u} = 2 u$ , $\frac{\partial z}{\partial v} = - 2 v$ . Why this step? $z = u^{2} - v^{2}$ ; note the sign on the $v$ term.

Step 2. $\frac{\partial u}{\partial x} = 1$ , $\frac{\partial v}{\partial x} = 1$ . Why this step? The two roads $x \to u$ and $x \to v$ each have slope $1$ .

Step 3. Sum: $\frac{\partial z}{\partial x} = 2 u \cdot 1 + (- 2 v) \cdot 1 = 2 (x + y) - 2 (x - y) = 4 y .$ Why this step? One path pushes up, the other pulls down; the $x$ -terms cancel.

Step 4. At $(0, 5)$ : $4 y = 20$ .

Step 5 — the exact-cancel case. Now take $(x, y) = (7, 0)$ and compute each single path's product explicitly, so we can see them cancel:

Here $u = x + y = 7$ and $v = x - y = 7$ .
$u$ -path product: $\frac{\partial z}{\partial u} \cdot \frac{\partial u}{\partial x} = (2 u) (1) = 2 \cdot 7 = + 14$ .
$v$ -path product: $\frac{\partial z}{\partial v} \cdot \frac{\partial v}{\partial x} = (- 2 v) (1) = - 2 \cdot 7 = - 14$ .
Sum: $+ 14 + (- 14) = 0$ .

Why this step? It shows the total can be $0$ while each individual path carries a hefty $\pm 14$ — the paths are alive but exactly opposite.

Verify: Direct: $z = (x + y)^{2} - (x - y)^{2} = 4 x y$ , so $\partial z / \partial x = 4 y$ . At $(0, 5)$ : $20$ . ✓ At $(7, 0)$ : $0$ , and the two path products are $+ 14$ and $- 14$ . ✓ Lesson: never assume a nonzero path means a nonzero total.

Example 4 — Cell D: a degenerate/zero link kills a path

Worked example A path with a zero link

Given: $z = u + v^{2}$ , $u = x^{3}$ , $v = y$ (so $v$ does not depend on $x$ ). Find $\frac{\partial z}{\partial x}$ .

Forecast: There look like two intermediates, but does $x$ actually reach $z$ through $v$ ?

Step 1. $\frac{\partial z}{\partial u} = 1$ , $\frac{\partial z}{\partial v} = 2 v$ . Why this step? These are the two link derivatives out of $z$ : from $z = u + v^{2}$ , the coefficient of $u$ is $1$ and differentiating $v^{2}$ gives $2 v$ . We need them before we can walk either path.

Step 2. $\frac{\partial u}{\partial x} = 3 x^{2}$ , and crucially $\frac{\partial v}{\partial x} = 0$ . Why this step? $v = y$ contains no $x$ , so the link $x \to v$ has zero strength — that whole path is dead.

Step 3. Sum: $\frac{\partial z}{\partial x} = 1 \cdot 3 x^{2} + 2 v \cdot 0 = 3 x^{2} .$ Why this step? A zero factor anywhere on a path zeroes that path's entire product. Include it consciously — don't just "forget" it.

Verify: $z = x^{3} + y^{2}$ directly, $\partial z / \partial x = 3 x^{2}$ . ✓ The $v$ -path contributed nothing, exactly as the diagram predicts.

Example 5 — Cell E: the neural neuron (saturation shrinks the gradient)

Worked example One neuron, backprop, saturated case

Given: $h = σ (a)$ with $a = w x + b$ , and loss $L = \frac{1}{2} (h - t)^{2}$ . Take $w = 2, x = 3, b = 1, t = 0$ . Find $\frac{\partial L}{\partial w}$ and comment on its size.

Forecast: $a = 2 \cdot 3 + 1 = 7$ is large, so $h = σ (7)$ is very close to $1$ . Do you expect a big or tiny gradient?

Step 1. Build the path $w \to a \to h \to L$ . Link derivatives: $\frac{\partial L}{\partial h} = h - t, \frac{\partial h}{\partial a} = h (1 - h), \frac{\partial a}{\partial w} = x .$ Why this step? Each is one arrow: loss slope, sigmoid slope, and $a$ 's slope in $w$ .

Step 1b — prove the sigmoid slope $σ^{'} (a) = σ (a) (1 - σ (a))$ by hand. Write $σ (a) = (1 + e^{- a})^{- 1}$ . By the (single-variable) chain rule, $σ^{'} (a) = - (1 + e^{- a})^{- 2} \cdot \frac{d}{d a} (1 + e^{- a}) = - (1 + e^{- a})^{- 2} \cdot (- e^{- a}) = \frac{e ^{- a}}{( 1 + e ^{- a} ) ^{2}} .$ Now split the fraction: $\frac{e ^{- a}}{( 1 + e ^{- a} ) ^{2}} = \frac{1}{1 + e ^{- a}} \cdot \frac{e ^{- a}}{1 + e ^{- a}}$ . The first factor is $σ (a)$ ; the second is $\frac{e ^{- a}}{1 + e ^{- a}} = \frac{( 1 + e ^{- a} ) - 1}{1 + e ^{- a}} = 1 - σ (a)$ . Hence $σ^{'} (a) = σ (a) (1 - σ (a))$ , i.e. $h (1 - h)$ . (Machine-checked in VERIFY.) Why this step? This is the promised proof — it turns "standard result" into something you derived.

Step 2. Multiply along the single path: $\frac{\partial L}{\partial w} = (h - t) h (1 - h) x .$

Step 3. Plug numbers. $a = 7 \Rightarrow h = σ (7) \approx 0.99909$ . Then $h (1 - h) \approx 0.000911$ . $\frac{\partial L}{\partial w} \approx (0.99909 - 0) \cdot 0.000911 \cdot 3 \approx 0.002729.$ Why this step? The factor $h (1 - h)$ is the sigmoid's slope; near $h = 1$ it is nearly zero — the neuron is saturated, so almost no gradient flows back. This is the seed of Vanishing and Exploding Gradients.

Verify: Numeric check in VERIFY confirms $\approx 0.00273$ . Sanity: the neuron is stuck near $1$ , learning slowly — exactly why saturated units learn poorly. ✓

Study the figure before reading on. The blue S-curve is the sigmoid $h$ ; the yellow bump is its slope $h (1 - h)$ — the exact factor from Step 1b that decides how much gradient survives. Notice the yellow curve peaks at the green dot ( $a = 0, h = 0.5$ , slope $= 0.25$ , the "active learning zone") and collapses toward the red dot ( $a = 7, h \approx 0.999$ , slope $\approx 0.0009$ , the "saturated" case of this very example). The tiny gradient we computed ( $\approx 0.0027$ ) is small precisely because our neuron sits at the red dot on the yellow curve — that is the visual meaning of a vanishing gradient.

Figure 2 — Sigmoid (blue) and its slope $h (1 - h)$ (yellow): steep at the centre (green), nearly flat when saturated (red).

Example 6 — Cell F: a real-world rates-over-time word problem

Worked example Expanding metal plate (chain rule in time)

Given: A heated rectangular plate has width $u$ and height $v$ ; its area is $z = uv$ . As it heats, $u$ grows at $\frac{d u}{d t} = 0.2$ cm/s and $v$ grows at $\frac{d v}{d t} = 0.1$ cm/s. At the instant $u = 10$ cm, $v = 6$ cm, how fast is the area growing?

Forecast: Time $t$ is the single input; it reaches area through two intermediates $u$ and $v$ . Two paths ⇒ add. Guess: a few cm²/s?

Step 1. Identify the graph: $t \to u \to z$ and $t \to v \to z$ . Why this step? Time changes both side lengths, and both feed the area.

Step 2. Link derivatives: $\frac{\partial z}{\partial u} = v$ , $\frac{\partial z}{\partial v} = u$ . Why this step? Area $z = uv$ .

Step 3. Multivariate chain rule in $t$ (sum the two paths): $\frac{d z}{d t} = \frac{\partial z}{\partial u} \frac{d u}{d t} + \frac{\partial z}{\partial v} \frac{d v}{d t} = v \cdot 0.2 + u \cdot 0.1.$

Step 4. Plug $u = 10, v = 6$ : $\frac{d z}{d t} = 6 (0.2) + 10 (0.1) = 1.2 + 1.0 = 2.2$ cm²/s.

Verify: Units: $(cm) (cm/s) = cm^{2} / s$ — an area rate, correct. Numeric $2.2$ cm²/s confirmed in VERIFY. ✓

Example 7 — Cell G: vector/Jacobian, the exam order-trap

Worked example Composing two layers — get the multiply order right

Given: $u = A x$ then $z = B u$ , with $A = [1021], B = [1301] .$ Find the Jacobian $\frac{\partial z}{\partial x}$ .

Forecast: The chain rule says $\frac{\partial z}{\partial x} = \frac{\partial z}{\partial u} \frac{\partial u}{\partial x}$ . The exam trap: is it $B A$ or $A B$ ? (Matrix multiply does not commute.) Guess before computing.

Step 1. $\frac{\partial z}{\partial u} = B$ (since $z = B u$ ), $\frac{\partial u}{\partial x} = A$ (since $u = A x$ ). Why this step? The Jacobian of a linear map $M x$ is just $M$ — each output is a fixed linear combination of the inputs, so its table of partial derivatives is $M$ itself.

Step 2. Chain rule keeps the outer-then-inner order: the outer Jacobian $\frac{\partial z}{\partial u} = B$ goes on the left, the inner $\frac{\partial u}{\partial x} = A$ on the right, so $\frac{\partial z}{\partial x} = B A .$ Why this step? Entry $(i, j)$ of the answer must be $\sum_{k} \frac{\partial z _{i}}{\partial u _{k}} \frac{\partial u _{k}}{\partial x _{j}}$ — that is exactly "row $i$ of $B$ dotted with column $j$ of $A$ ," which is the definition of the matrix product $B A$ (in that order). Writing $A B$ would dot the wrong rows and columns together (see Jacobian and Hessian Matrices; this ordering is the heart of Backpropagation Algorithm).

Step 3. Compute: $B A = [1301] [1021] = [1327] .$ Why this step? Row 1 of $B$ is $(1, 0)$ : dotted with the columns of $A$ gives $(1, 2)$ . Row 2 of $B$ is $(3, 1)$ : dotted with the columns of $A$ gives $(3 \cdot 1 + 1 \cdot 0, 3 \cdot 2 + 1 \cdot 1) = (3, 7)$ .

Verify: Direct: substituting, $z = B (A x) = (B A) x$ , so its Jacobian is $B A$ . Also $A B = [7321] \neq = B A$ — proving the order genuinely matters. Both facts confirmed in VERIFY. ✓

Example 8 — Cell H: limiting behaviour (why deep saturated nets stall)

Worked example The limit of stacked saturated slopes

Given: $L$ passes back through $n$ sigmoid neurons in series; each contributes a slope factor at most $max_{h} h (1 - h)$ . What is the largest single-neuron slope, and what happens to the product through $n$ layers as $n \to \infty$ ?

Forecast: Sigmoid's slope $h (1 - h)$ is a downward parabola in $h \in (0, 1)$ . Where is it biggest, and how big? Then guess the fate of that number raised to the $n$ .

Step 1. Maximise $s (h) = h (1 - h)$ . Set $s^{'} (h) = 1 - 2 h = 0 \Rightarrow h = 0.5$ . Why this step? The steepest the sigmoid ever is happens at its centre $h = 0.5$ ; setting the derivative of the slope to zero finds that peak.

Step 2. Value there: $s (0.5) = 0.5 \cdot 0.5 = 0.25$ . So every sigmoid link multiplies the backward gradient by at most $0.25$ . Why this step? This ceiling of $0.25$ is what turns "one layer loses a little gradient" into "many layers lose almost all of it."

Step 3. Through $n$ layers the gradient is scaled by at most $0.2 5^{n}$ : $lim_{n \to \infty} 0.2 5^{n} = 0.$ Why this step? A number below $1$ raised to a growing power collapses to $0$ — the vanishing gradient. This is the limiting scenario that motivates Activation Functions like ReLU and links to Gradient Descent slowing to a crawl.

Verify: $s (0.5) = 0.25$ and $0.2 5^{10} \approx 9.5 \times 1 0^{- 7}$ (already microscopic) confirmed in VERIFY. ✓

Recall Quick self-test

Two paths from $x$ to $z$ — combine how? ::: Multiply along each path, then add the paths. A link with derivative $0$ does what to its path? ::: Kills the whole path (product becomes $0$ ). Biggest possible sigmoid slope $h (1 - h)$ ? ::: $0.25$ , at $h = 0.5$ . In $\partial z / \partial x = ?$ for $z = B u, u = A x$ ? ::: $B A$ (outer Jacobian on the left). Units of $d z / d t$ when $z$ is area (cm²)? ::: cm²/s.

Related vault notes: Computational Graphs · Automatic Differentiation · Loss Functions.