Visual walkthrough — Chain rule for multivariate functions (backprop foundation)

Nothing here assumes you already know derivatives as formulas. We earn every symbol before using it.

---

Step 1 — What a derivative even means: a wiggle-ratio

WHAT. Suppose one number, call it $y$, depends on another number $x$. Picture a machine: you push a dial to value $x$, a needle points to value $y$. Now nudge the dial by a tiny extra amount. We write that tiny nudge as $\Delta x$ (read "delta-x" = "a small change in $x$"). The needle moves by a tiny amount $\Delta y$.

WHY. We want a single number that answers: "per unit of dial-nudge, how far does the needle move?" That number is the ratio $\frac{\Delta y}{\Delta x}$. As the nudge shrinks toward zero, this ratio settles on a fixed value — we name it $\frac{dy}{dx}$, the derivative. It is just the steepness of the machine's response.

$\frac{dy}{dx}=\underset{\text{output wiggle}÷\text{input wiggle}}{\underbrace{\frac{\Delta y}{\Delta x}}}\text{as }\Delta x\to 0$

PICTURE. The red line is the curve $y(x)$. Zoom into one point and the curve looks straight; its slope is the wiggle-ratio.

---

Step 2 — Chaining two machines: multiply the wiggle-ratios

WHAT. Now put two machines in a line. Machine 1 turns $x$ into a middle quantity $u$. Machine 2 turns $u$ into $y$. So $x\to u\to y$.

WHY multiply? Follow one nudge. Nudge $x$ by $\Delta x$. Machine 1 has steepness $\frac{du}{dx}$, so $u$ moves by $\Delta u\approx \frac{du}{dx}\Delta x$. That $\Delta u$ enters Machine 2, which has steepness $\frac{dy}{du}$, so $y$ moves by $\Delta y\approx \frac{dy}{du}\Delta u$. Substitute:

$\Delta y\approx \underset{\text{gain of machine 2}}{\underbrace{\frac{dy}{du}}}\underset{\text{gain of machine 1}}{\underbrace{\frac{du}{dx}}}\Delta x$

Divide by $\Delta x$: the two gains multiply. That is the single-variable chain rule.

$\frac{dy}{dx}=\frac{dy}{du}\cdot \frac{du}{dx}$

PICTURE. Two gear-boxes in series. A small twist $\Delta x$ becomes $\Delta u$ through the first gain, then $\Delta y$ through the second — the red arrow is the one signal path.

---

Step 3 — Two inputs into one machine: the total differential

WHAT. Let the final machine take two inputs at once: $z=f(u,v)$. Think of a landscape where height $z$ depends on east-position $u$ and north-position $v$.

WHY two derivatives now? With two dials there are two independent steepnesses:

• $\frac{\partial z}{\partial u}$ — how height changes if you step east holding north fixed.
• $\frac{\partial z}{\partial v}$ — how height changes if you step north holding east fixed.

The curly $\partial$ ("partial") just means "wiggle-ratio while freezing the other inputs." A tiny step that moves both dials at once changes height by the sum of the two separate effects:

$dz=\underset{\text{east steepness}}{\underbrace{\frac{\partial z}{\partial u}}}du+\underset{\text{north steepness}}{\underbrace{\frac{\partial z}{\partial v}}}dv$

This is the total differential — the total height change is east-slope times east-step plus north-slope times north-step.

PICTURE. A tilted plane (the local surface) over the $u$–$v$ ground. Moving $du$ east climbs one amount, $dv$ north climbs another; total rise is their sum (red).

---

Step 4 — Both middle-men trace back to the same input

WHAT. Now suppose both $u$ and $v$ are themselves produced from the same input $x$: $u=u(x)$ and $v=v(x)$. So the wiring is a diamond: $x$ splits into two, feeds $u$ and $v$, and they rejoin at $z$.

WHY this is the whole point. When $x$ nudges by $\Delta x$, it wiggles $u$ and $v$ at the same time:

$du=\frac{\partial u}{\partial x}dx,dv=\frac{\partial v}{\partial x}dx$

Each middle-man carries the same $dx$ forward through its own gain.

PICTURE. The diamond graph: one source node $x$, two middle nodes $u,v$, one sink $z$. The red edges are the two paths a nudge can travel.

---

Step 5 — Substitute: the two paths ADD

WHAT. Take the total differential from Step 3 and pour in the $du,dv$ from Step 4:

= \frac{\partial z}{\partial u}\Big(\tfrac{\partial u}{\partial x}dx\Big) + \frac{\partial z}{\partial v}\Big(\tfrac{\partial v}{\partial x}dx\Big)$$ **WHY.** Factor out the common $dx$: $$dz = \left(\;\underbrace{\frac{\partial z}{\partial u}\frac{\partial u}{\partial x}}_{\text{path through }u} \;+\; \underbrace{\frac{\partial z}{\partial v}\frac{\partial v}{\partial x}}_{\text{path through }v}\;\right)dx$$ The number multiplying $dx$ *is* the wiggle-ratio $\dfrac{\partial z}{\partial x}$. Read it off: > [!formula] Multivariate chain rule (two paths) > $$\frac{\partial z}{\partial x} = \underbrace{\frac{\partial z}{\partial u}\frac{\partial u}{\partial x}}_{\text{multiply along path 1}} + \underbrace{\frac{\partial z}{\partial v}\frac{\partial v}{\partial x}}_{\text{multiply along path 2}}$$ > **Rule in words:** ==multiply the gains along each path, then add all paths.== **PICTURE.** Two colored routes from $x$ to $z$; each route's total gain is the product of its edges; the answer is their sum. ![[deepdives/dd-ai-ml-1.2.14-d2-s05.png]] --- ## Step 6 — Many paths: the sum grows, the rule stays **WHAT.** If $z$ depends on $n$ middle variables $u_1,\dots,u_n$, and every $u_i$ depends on $x$, there are now $n$ parallel routes. **WHY.** Nothing changed except *how many* paths rejoin. Each path still contributes (gain in) $\times$ (gain out), and they all add: $$\frac{\partial z}{\partial x_j} = \sum_{i=1}^{n} \frac{\partial z}{\partial u_i}\,\frac{\partial u_i}{\partial x_j}$$ Every symbol: $x_j$ is one input dial, $u_i$ is the $i$-th middle quantity, $\dfrac{\partial z}{\partial u_i}$ is how strongly middle-man $i$ pushes the output, $\dfrac{\partial u_i}{\partial x_j}$ is how strongly the input pushes middle-man $i$. Sum over all $i$ = visit every route. **PICTURE.** A fan of $n$ paths from one input, each labelled with its two-factor gain, all feeding the sum node. ![[deepdives/dd-ai-ml-1.2.14-d2-s06.png]] > [!intuition] Why this is exactly backprop > Reverse the arrows and you get [[Backpropagation Algorithm|backprop]]: start with $\dfrac{\partial L}{\partial z}=1$ at the output, and push it backward, multiplying by each local gain and **summing where paths merge**. See also [[Automatic Differentiation]] and [[Jacobian and Hessian Matrices]]. --- ## Step 7 — Degenerate & edge cases (so nothing surprises you) **WHAT / WHY, case by case:** - **A path that doesn't exist.** If $v$ does *not* depend on $x$, then $\dfrac{\partial v}{\partial x}=0$ and that whole term vanishes — the sum automatically drops it. The formula never breaks; it just zeroes dead wires. - **A single path only.** If there's just one middle-man, the sum has one term and we recover the Step-2 single-variable rule. Multivariate is the *general* case; single-variable is a special case. - **Saturated gain (zero derivative in the middle).** If one link has gain $0$ (e.g. a flat activation region, $h(1-h)\to 0$ for a saturated sigmoid), the *entire product on that path* is $0$. Signal cannot pass — this is precisely [[Vanishing and Exploding Gradients|the vanishing gradient]] story. Choice of [[Activation Functions]] controls this. - **Exploding gain.** If many stacked gains each exceed $1$, their product blows up — the same mechanism running the other way. - **Shared vs. independent inputs.** For a *second* input $y$, repeat the whole thing with $\dfrac{\partial u}{\partial y}$ and $\dfrac{\partial v}{\partial y}$ — same shape, different letters. **PICTURE.** The diamond with the $v$-branch cut (grey, gain $0$) — the surviving red path is all that carries signal. ![[deepdives/dd-ai-ml-1.2.14-d2-s07.png]] --- ## Worked check (numbers, so you trust the pictures) > [!example] The parent's Example 1, re-seen as paths > $z=u^2+v^2$, $u=2x+y$, $v=x-3y$. Path gains: > $$\frac{\partial z}{\partial u}=2u,\;\frac{\partial u}{\partial x}=2 \quad\Rightarrow\quad \text{path-}u = 4u$$ > $$\frac{\partial z}{\partial v}=2v,\;\frac{\partial v}{\partial x}=1 \quad\Rightarrow\quad \text{path-}v = 2v$$ > Add the paths: $\dfrac{\partial z}{\partial x}=4u+2v=4(2x+y)+2(x-3y)=10x-2y$. ✓ > Direct check: $z=5x^2-2xy+10y^2\Rightarrow \dfrac{\partial z}{\partial x}=10x-2y$. Same answer. --- ## The one-picture summary Everything above collapses into one image: **the gain of a path is the product of its edges; the derivative w.r.t. an input is the sum over all paths that leave it.** Multiply along, add across. ![[deepdives/dd-ai-ml-1.2.14-d2-s08.png]] > [!recall]- Feynman retelling — say it back in plain words > A derivative is just a *wiggle-ratio*: push the input a hair, see how far the output moves, divide. When machines sit in a line, their wiggle-ratios **multiply** — like stacked amplifiers. When one input feeds several middle-men that all rejoin at the output, you get several routes; each route's total effect is (input→middle gain) times (middle→output gain), and because the little pushes all arrive together, the routes **add**. Write that "multiply-along, add-across" rule once and you have: the single-variable chain rule (one route), backpropagation (routes run backward), the vanishing gradient (a route with a zero gain kills its whole product), and the Jacobian product (many inputs and many outputs at once). One idea, drawn as a graph, powers all of deep-learning's [[Gradient Descent|gradient descent]]. > [!recall]- Quick self-test > Why do path contributions get summed, not multiplied? ::: Because the input nudges every middle variable *at the same time*; their separate effects on the output add up (total differential). > Why do the two factors *within* one path get multiplied? ::: Each factor is a local gain; a nudge passes through them in series, so gains multiply — like amplifiers in a line. > What happens to a path whose middle link has derivative $0$? ::: Its product becomes $0$; the path carries no signal (vanishing gradient). > If $v$ does not depend on $x$, what happens to its term? ::: $\partial v/\partial x=0$, so that whole term drops out automatically.