Foundations — Chain rule for multivariate functions (backprop foundation)

This page assumes you know nothing about the notation in the parent note. We will earn every squiggle before it is allowed to appear. If a symbol shows up in the parent's formulas, it gets a plain-words meaning, a picture, and a reason for existing, right here.

0. What is a function, and why the arrows?

Before any calculus, fix the picture in your head: a function is a machine. You drop a number in, one number comes out.

The picture: a box labelled $f$. An arrow goes in carrying $x$, an arrow goes out carrying $z$. When machines are wired in a line — the output of one becomes the input of the next — we get a chain, and that chain is the entire reason this topic exists.

Cloze check: a function returns exactly one output for each input.

1. The symbols $x,y,z,u,v$ — names for travellers

These are just labels for numbers that flow through the machines.

• $x,y$ — the inputs we control (the knobs at the far left).
• $u,v$ — intermediate values, produced partway along, before the end.
• $z$ — the final output (in ML this becomes the loss $L$).

The picture: stations on an assembly line. $x$ is raw material, $u$ and $v$ are half-built parts, $z$ is the finished product.

Why the topic needs them: the chain rule is about tracing influence from an input all the way to the output. You cannot trace a path without naming the stops along it. $u$ and $v$ are the stops.

2. Slope — "how sensitive is out to in?"

Here is the seed of everything. Take one machine $z=f(x)$. Nudge the input by a tiny amount. How much does the output move?

The picture: the graph of $f$ is a curve. Zoom in on one point until the curve looks like a straight line — a ramp. Slope = rise ÷ run of that ramp: go right by "run", the line climbs by "rise". A steep ramp = big sensitivity; a flat ramp = tiny sensitivity.

Why this tool and not another? We could describe a curve a thousand ways, but the only thing that matters for "how does a nudge propagate" is the local steepness. Slope is precisely the sensitivity number, and sensitivity is exactly what backpropagation shuttles backward.

3. The tiny-change symbol $\Delta$ and the limit

The parent writes things like $\Delta x$ and "take the limit". Let's earn those.

The picture: on the curve from §2, put two dots a distance $\Delta x$ apart. The line through them is a rough ramp. Slide the second dot towards the first ($\Delta x\to 0$); the rough ramp snaps onto the true tangent ramp. The number it settles on is the exact slope.

Why we need the limit: for a straight ramp, rise/run is the same everywhere. For a curve, rise/run changes depending on how big your step is — until you shrink the step to nothing. The limit removes that ambiguity and gives one honest slope per point.

4. The derivative $\frac{dz}{dx}$ — the settled slope, given a name

That settled-down slope gets its own symbol.

The picture: the exact tangent ramp from §3, and $\frac{dz}{dx}$ is its rise-over-run.

Why the topic needs it: every "how sensitive is this link?" number in the chain rule is a derivative. The chain is a row of these tangent ramps handed one to the next.

5. Functions of two inputs, and the partial $\partial$

Real machines have several knobs: $z=f(u,v)$ depends on two inputs at once. Now "the slope" is ambiguous — slope as you push which knob?

The picture: the graph of $f(u,v)$ is now a surface — a hillside — sitting over a flat $u$–$v$ floor. Walk east (increase $u$, keep $v$ fixed): the steepness of the ground under your east-facing step is $\frac{\partial z}{\partial u}$. Turn and walk north (increase $v$, keep $u$ fixed): that steepness is $\frac{\partial z}{\partial v}$. Same hill, two different slopes depending on your compass heading.

Why this tool and not the ordinary derivative? A neural network's output depends on thousands of weights simultaneously. To ask "how does the loss react to this one weight," we must freeze all the others — that is exactly what $\partial$ enforces. Ordinary $d$ can't express "hold the rest still."

6. Multiple paths — why we ADD

This is the heart of the multivariate case, and the parent's most important idea.

Suppose one input $x$ affects the output $z$ through two different routes: $x\to u\to z$ and $x\to v\to z$. When you nudge $x$, both routes carry a wave to $z$ at the same time.

The picture: a river splits into two channels around an island and rejoins. Water you pour in at the top ($x$) arrives at the bottom ($z$) via both channels; the total flow at the bottom is the sum of the two channels.

• Along one channel you MULTIPLY: the nudge in $x$ becomes a nudge in $u$ (scaled by $\frac{\partial u}{\partial x}$), which becomes a nudge in $z$ (scaled by $\frac{\partial z}{\partial u}$). Two scalings in a row = multiply.
• Across channels you ADD: because the two effects on $z$ arrive independently and pile up.

7. The differential $dz$ — the "sum of channels" written cleanly

The parent uses $dz=\frac{\partial f}{\partial u}du+\frac{\partial f}{\partial v}dv$. This is just §6 said in one line before choosing which input to push.

Why it exists: it lets us postpone the question "with respect to what?" We collect all contributions first, then afterwards divide by $dx$ (holding $y$ fixed) to read off $\frac{\partial z}{\partial x}$. It is bookkeeping that keeps every path in view.

8. The sum sign $\sum$ — many paths at once

When there are $n$ intermediate stations $u_1,u_2,\dots ,u_n$, writing every path by hand is silly. The symbol $\sum$ ("sigma", Greek S for Sum) means "add up the following as the counter runs through its range."

Read it aloud: "for input number $j$, add up — over every intermediate $u_i$ — the slope from $x_j$ into $u_i$ times the slope from $u_i$ into $z$." It's §6 with as many channels as you like.

9. Vectors, matrices, and the Jacobian — packaging the sums

The parent's Example 3 jumps to $\mathbf{z},\mathbf{u},\mathbf{x}$ in bold and calls a table of partials a Jacobian. Bold letters just mean a list of numbers stacked together (a vector). A grid of numbers is a matrix.

Why bother? The chain-rule sum ${\sum }_k\frac{\partial z_i}{\partial u_k}\frac{\partial u_k}{\partial x_j}$ is exactly the recipe for matrix multiplication (row dotted with column). So stacking the partials into grids lets one clean product $\frac{\partial \mathbf{z}}{\partial \mathbf{u}}\frac{\partial \mathbf{u}}{\partial \mathbf{x}}$ replace thousands of hand-written sums. That is why Jacobians are the natural language of layer-by-layer backprop. (Full detail is left to those notes — here you only need to know the word names a table of partials.)

10. The sigmoid $\sigma$ and its neat slope

Example 2 uses $\sigma (a)=\frac{1}{1+e^{-a}}$. You don't need its internals yet, only two facts:

• $\sigma$ is a squashing machine: any input, output between $0$ and $1$. It is one of the Activation Functions.
• Its slope has a famously tidy form: ${\sigma }^{\prime }(a)=\sigma (a)(1-\sigma (a))$. In the chain, this is just "the link's sensitivity number" and drops straight into the product.

We flag it here so the symbol $\sigma$ is never a surprise; the derivative itself is derived in the Activation Functions note.

Prerequisite map

This map feeds directly into the parent topic, which in turn powers Backpropagation Algorithm, Computational Graphs, Automatic Differentiation and Gradient Descent.

Equipment checklist

Test yourself — if you can complete each reveal from memory, you are ready for the parent note.