Chain rule for multivariate functions (backprop foundation)

What Is the Multivariate Chain Rule?

WHY does this work? Because when $x$ changes by a tiny amount $\Delta x$:

1. It causes $u$ to change by $\approx \frac{\partial u}{\partial x} \Delta x$
2. That change in $u$ causes $z$ to change by $\approx \frac{\partial z}{\partial u} \cdot \frac{\partial u}{\partial x} \Delta x$
3. Simultaneously, $x$ also affects $v$, which affects $z$ through the second path
4. The total change in $z$ is the sum of both pathway contributions

Derivation from First Principles

Let's build this from the ground truth of limits and differentials.

Step 1: Start with the single-variable chain rule For $y = f(u)$ and $u = g(x)$, we know: $\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$

WHY? Because $\Delta y \approx \frac{dy}{du} \Delta u$ and $\Delta u \approx \frac{du}{dx} \Delta x$, so: $\Delta y \approx \frac{dy}{du} \cdot \frac{du}{dx} \cdot \Delta x$

Dividing by $\Delta x$ and taking the limit gives chain rule.

Step 2: Generalize to multiple intermediate variables Now let $z = f(u, v)$ where both $u$ and $v$ depend on $x$.

From the definition of partial derivatives: $dz = \frac{\partial f}{\partial u} du + \frac{\partial f}{\partial v} dv$

This is the total differential of $f$. It says: "A tiny change in $z$ comes from contributions through $u$ and through $v$."

Step 3: Express $du$ and $dv$ in terms of $dx$ $du = \frac{\partial u}{\partial x} dx + \frac{\partial u}{\partial y} dy$ $dv = \frac{\partial v}{\partial x} dx + \frac{\partial v}{\partial y} dy$

Step 4: Substitute and collect terms $dz = \frac{\partial f}{\partial u}\left(\frac{\partial u}{\partial x} dx + \frac{\partial u}{\partial y} dy\right) + \frac{\partial f}{\partial v}\left(\frac{\partial v}{\partial x} dx + \frac{\partial v}{\partial y} dy\right)$

Collecting coefficients of $dx$: $dz = \left(\frac{\partial f}{\partial u} \frac{\partial u}{\partial x} + \frac{\partial f}{\partial v} \frac{\partial v}{\partial x}\right) dx + \left(\frac{\partial f}{\partial u} \frac{\partial u}{\partial y} + \frac{\partial f}{\partial v} \frac{\partial v}{\partial y}\right) dy$

By definition, the coefficient of $dx$ is $\frac{\partial z}{\partial x}$:

WHAT does this mean? Each term in the sum is one "pathway" from input $x_j$ through intermediate $u_i$ to output $z$. We multiply along each path and sum all paths.

Worked Examples

Connection to Backpropagation

In neural networks, we have a computational graph where:

• Nodes = intermediate values (activations, weighted sums)
• Edges = operations (matrix multiply, activation functions)
• Goal = compute $\frac{\partial L}{\partial w}$ for every parameter $w$

The backprop algorithm IS the chain rule:

1. Forward pass: Compute $z$ from $x$ through all intermediates, storing values
2. Backward pass: Compute $\frac{\partial L}{\partial x}$ by propagating gradients backward through the graph

At each node, if $z$ depends on $u_1, \ldots, u_n$: $\frac{\partial L}{\partial u_i} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial u_i}$

This is called the local gradient ($\frac{\partial z}{\partial u_i}$) times the upstream gradient ($\frac{\partial L}{\partial z}$).

WHY is this efficient? Instead of computing each $\frac{\partial L}{\partial w_i}$ independently (expensive), we compute one backward pass that gives us ALL parameter gradients. This is reverse-mode automatic differentiation.

Why This Matters for AI/ML

1. Backpropagation is just repeated chain rule application. Every neural network training relies on this.

2. Computational graphs make it visual. Modern frameworks (PyTorch, TensorFlow) build a graph and automatically apply the chain rule.

3. Efficiency: Without the chain rule, computing gradients would require $O(n)$ forward passes for $n$ parameters. With it, one backward pass gives all gradients.

4. Vanishing/exploding gradients happen when chain rule products become very small/large through many layers: $\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial z_n} \cdot \frac{\partial z_n}{\partial z_{n-1}} \cdots \frac{\partial z_2}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_1}$ If each $\frac{\partial z_i}{\partial z_{i-1}} < 1$, the product shrinks exponentially (vanishing). If $> 1$, it explodes.

Connections

• Gradient Descent - Uses these gradients to update parameters
• Backpropagation Algorithm - The practical implementation of chain rule in neural nets
• Computational Graphs - Visual representation enabling automatic differentiation
• Automatic Differentiation - Software that computes chain rule automatically
• Jacobian and Hessian Matrices - Matrix forms of multivariate derivatives
• Vanishing and Exploding Gradients - Problems arising from long chains
• Activation Functions - Each has a derivative used in the chain
• Loss Functions - Starting point ($\frac{\partial L}{\partial y}$) for backprop

#flashcards/ai-ml

What is the multivariate chain rule formula for $\frac{\partial z}{\partial x}$ when $z = f(u,v)$ with $u=g(x,y)$ and $v=h(x,y)$? :: $\frac{\partial z}{\partial x} = \frac{\partial z}{\partial u}\frac{\partial u}{\partial x} + \frac{\partial z}{\partial v}\frac{\partial v}{\partial x}$. Sum over all paths from $x$ to $z$, multiplying derivatives along each path.

Why do we SUM the terms in the multivariate chain rule? :: Because the input variable (e.g., $x$) can affect the output through MULTIPLE intermediate variables (multiple paths). Each path contributes independently to the total change, so we add them.

What's the difference between $\frac{\partial z}{\partial x}$ and $\frac{dz}{dx}$? :: $\frac{\partial z}{\partial x}$ is the partial derivative holding other variables constant. $\frac{dz}{dx}$ is the total derivative accounting for ALL ways $x$ affects $z$, including indirect paths through other variables.

Concept Map

Hinglish (regional understanding)