Chain rule for multivariate functions (backprop foundation): Question bank

Before we start, one shared vocabulary reminder so nothing below uses an unexplained symbol:

Keep this little map in your mind's eye as you work — most items are really asking "which arrows exist, and do I add or multiply along them?"

Figure — Chain rule for multivariate functions (backprop foundation)

True or false — justify

z

depends on

x

only through a single intermediate

u

, the multivariate chain rule reduces to a plain product

\frac{\partial z}{\partial u} \frac{\partial u}{\partial x}

True — with only one path, the "sum over paths" has a single term, so the sum collapses to that one product. The multivariate rule always contains the single-variable rule as a special case.

The chain rule

\frac{\partial z}{\partial x} = \sum_{i} \frac{\partial z}{\partial u _{i}} \frac{\partial u _{i}}{\partial x}

requires the

u_{i}

to be independent of each other.

False — the

u_{i}

can freely depend on the same inputs; that's exactly why we sum. Independence is never assumed. What matters is that each

u_{i}

is a direct function argument of

z

\frac{\partial z}{\partial u} = 0

for one intermediate, that whole path contributes nothing to

\frac{\partial z}{\partial x}

True — a zero factor kills the product for that path (multiply-along-the-path), so it drops out of the sum. The other paths can still carry gradient. This is exactly how a saturated neuron blocks one route but not necessarily all.

Two different paths can cancel, giving

\frac{\partial z}{\partial x} = 0

even though

z

genuinely changes along each individual path.

True — path contributions are signed and add; a

+ 3

and a

- 3

sum to zero. So a zero gradient does not mean

x

is irrelevant everywhere, only that its net first-order effect cancels at that point.

The total differential

d z = \frac{\partial z}{\partial u} d u + \frac{\partial z}{\partial v} d v

is exact, not an approximation.

True as a differential relationship (it's the exact linear part). The approximation

Δ z \approx \dots

only appears once you replace differentials with finite nudges

Δ u, Δ v

; the error there is higher-order small.

\frac{\partial z}{\partial x}

for

z = f (u, v)

with

u = g (x, y)

and

v = h (x, y)

, we hold

v

constant.

False — this is the classic trap. We hold the other independent input

y

constant, but

v

itself changes because

v

depends on

x

. Freezing

v

would delete a real path and give the wrong answer.

For

z = x + u (x)

, the correct derivative is

\frac{\partial z}{\partial x} = u^{'} (x)

, since the chain rule handles the

u

term.

False — this misses the direct dependence. There are two arrows into

z

: one straight from

x

(contributing

1

) and one through

u

(contributing

u^{'} (x)

), so

\frac{\partial z}{\partial x} = 1 + u^{'} (x)

. Whenever a variable reaches the output both directly and via an intermediate, both paths are summed.

Backpropagation and the multivariate chain rule are two different algorithms.

False — backprop is the chain rule, organised so each shared sub-result is computed once and reused. It's an efficient bookkeeping of the same sum-over-paths.

For

y = w_{2} h + b_{2}

, the derivative

\frac{\partial y}{\partial h} = w_{2}

even though

h

was itself produced by earlier weights.

True — a partial derivative treats the local relationship only. How

h

was made is irrelevant to

\frac{\partial y}{\partial h}

; that history gets handled by the next factor

\frac{\partial h}{\partial a}

down the chain.

Spot the error

x

affects

z

through

u

and through

v

, so

\frac{\partial z}{\partial x} = \frac{\partial z}{\partial u} \frac{\partial u}{\partial x} \cdot \frac{\partial z}{\partial v} \frac{\partial v}{\partial x}

The paths should be added, not multiplied. You multiply along one path and sum across different paths, so the correct expression is

\frac{\partial z}{\partial x} = \frac{\partial z}{\partial u} \frac{\partial u}{\partial x} + \frac{\partial z}{\partial v} \frac{\partial v}{\partial x}

"For

z = x^{2} + v (x)

I differentiate the chain part and write

\frac{\partial z}{\partial x} = v^{'} (x)

The direct term

x^{2}

was dropped. Both the direct arrow and the arrow through

v

count, giving

\frac{\partial z}{\partial x} = 2 x + v^{'} (x)

. Missing the direct dependence is the single most common slip in mixed direct-plus-indirect expressions.

"The sigmoid derivative is

σ^{'} (a) = σ (a)

, so I plug in

h

directly."

Wrong —

σ^{'} (a) = σ (a) (1 - σ (a)) = h (1 - h)

. Forgetting the

(1 - h)

factor overstates the gradient massively when

h

is near 1 and is a common cause of "why won't my by-hand gradient match the framework."

"To get

\frac{\partial L}{\partial w _{1}}

I only need

\frac{\partial L}{\partial y}

and

\frac{\partial a}{\partial w _{1}}

; the middle links cancel."

Nothing cancels. The full chain is

\frac{\partial L}{\partial w _{1}} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial h} \cdot \frac{\partial h}{\partial a} \cdot \frac{\partial a}{\partial w _{1}}

— every middle link must be multiplied in, or you silently delete the layers between loss and weight.

"Since

\frac{\partial z}{\partial x}

collects the

d x

coefficient, the

d y

terms are errors I should ignore."

The

d y

terms aren't errors — they are

\frac{\partial z}{\partial y}

, a legitimate separate answer. You're computing one partial at a time; the

d y

coefficient is simply another result, not garbage.

"Multiplying Jacobians is different from the scalar chain rule, so my scalar intuition doesn't transfer."

The Jacobian product is the sum-over-paths written compactly: entry

(i, j)

\sum_{k} \frac{\partial z _{i}}{\partial u _{k}} \frac{\partial u _{k}}{\partial x _{j}}

. Matrix multiplication automatically performs the sum-over-intermediates. See Jacobian and Hessian Matrices.

"Order doesn't matter, so I can multiply the Jacobians in either order when composing layers."

Matrix multiplication is not commutative, and dimensions won't even line up in the wrong order. The output-layer Jacobian must sit on the outside; the composition order mirrors the forward flow reversed.

Why questions

Why do we sum over paths instead of taking, say, the largest path?

Because a small nudge in

x

physically pushes

z

through every route simultaneously, and those simultaneous pushes add up to the total change. Linear approximation of independent contributions is additive.

Why does backprop reuse intermediate gradients instead of recomputing each path from scratch?

Many paths share a common prefix (e.g. the same

\frac{\partial L}{\partial y}

). Computing shared factors once and propagating them backward turns exponential path enumeration into linear work — the core efficiency behind Automatic Differentiation.

Why does a saturated sigmoid (

h \approx 0

h \approx 1

) stall learning?

Its local factor

h (1 - h) \to 0

, so every gradient flowing through that node is scaled down toward zero, starving upstream weights. Multiply many such small factors across depth and you get vanishing gradients.

Why can't we compute

\frac{\partial L}{\partial w _{1}}

without first doing a forward pass?

The gradient factors like

h (1 - h)

and

(y - t)

depend on the actual values of the intermediates and output. Those values only exist after running the network forward on the input.

Why does the parent note's factory-line metaphor demand backward tracing rather than forward?

We want to know how each early station affected the final defect, so we start at the output error and carry blame upstream. Forward would tell us outputs from inputs; backward tells us sensitivities of the loss to each parameter — exactly what Gradient Descent needs.

Why does the choice of activation function change the chain-rule factors even when the network structure is identical?

Each activation contributes its own

\frac{\partial h}{\partial a}

factor. Swap sigmoid for ReLU and that factor changes from

h (1 - h)

to a

0

-or-

1

gate, reshaping which paths carry gradient.

Edge cases

If an input

x

does not appear in any path to

z

, what is

\frac{\partial z}{\partial x}

Exactly

0

— there are no path terms to sum, so the empty sum is zero. The variable is genuinely irrelevant to the output at first order.

What happens to the chain rule when an intermediate is used by the output twice (fan-out to two later nodes)?

You get two paths sharing the same early segment; their contributions add. In graph terms, a fan-out node's incoming gradient is the sum of gradients from all its consumers.

At a ReLU kink (

a = 0

), is the chain rule factor well-defined?

Strictly the derivative doesn't exist at exactly

0

(left slope

0

, right slope

1

). Frameworks pick a subgradient convention (usually

0

), which works because you almost never land exactly on the kink and the loss is still minimised in practice.

If the loss is flat around the current prediction (

\frac{\partial L}{\partial y} = 0

), what does every upstream gradient become?

All zero, since

\frac{\partial L}{\partial y}

is a common leading factor multiplied into every path back to the weights. No error signal means no update — the network sits still even if weights are "wrong" in a deeper sense.

What is

\frac{\partial z}{\partial x}

when

u

depends on

x

but

z

does not actually depend on

u

(dead intermediate)?

Zero through that route:

\frac{\partial z}{\partial u} = 0

kills the product. If

u

x

's only route to

z

, the whole partial is zero even though

u

visibly moves with

x

For a single input feeding a very deep chain of

n

near-saturated nodes, why does the gradient scale roughly like a product of

n

small numbers?

Because the chain rule multiplies one local factor per layer along the single path;

n

factors each below

1

shrink geometrically. This exponential decay is precisely the vanishing gradient phenomenon.

Recall Quick self-test before you leave

Sum or product across different paths? ::: Sum across paths, product along each path. In $\frac{\partial z}{\partial x}$ with $z = f (u, v)$ , do we hold $v$ fixed? ::: No — we hold the other independent input fixed; $v$ still varies through its own dependence on $x$ . For $z = x + u (x)$ , what is $\frac{\partial z}{\partial x}$ ? ::: $1 + u^{'} (x)$ — the direct arrow plus the arrow through $u$ . Is backprop a different algorithm from the chain rule? ::: No — it is the chain rule with shared factors computed once and reused.