Foundations — Backpropagation — chain rule, gradient computation

This page assumes nothing. If you have never seen a derivative, a vector, or the little Greek letters, start here and you will be able to read the parent note line by line.

0. What a function is (the atom of everything)

Picture a box with an arrow going in and an arrow coming out.

Why the topic needs it. A neural net is a stack of these boxes. The whole parent note writes $L=f_N(\dots f_2(f_1(x)))$ — that is just many boxes wired in a line, the output of one becoming the input of the next.

1. Slope, and why we need the derivative

The picture: zoom into any smooth curve far enough and it looks like a straight line. The steepness of that tiny line is the derivative. Look at the green tangent line in the figure — its steepness is the number $df/dx$.

Why the topic needs it. Every "local derivative" in backprop (like ${\sigma }^{\prime }(z_1)$) is one of these slopes. The whole algorithm is multiplying slopes.

2. The partial derivative $\partial$ (many knobs at once)

The picture: stand on a hillside. Walking due east has one slope; walking due north has another. Each direction is one partial derivative of the same hill.

Why the topic needs it. A net has millions of weights $w$. We want $\partial L/\partial w$ for each one — the sensitivity of the error to that single knob while the rest are frozen. That list of all partials is the gradient.

3. The gradient $\nabla$ (the whole bundle of slopes)

The picture: on the hillside, gather the east-slope and north-slope into a single arrow. That arrow aims straight uphill; its length says how steep. To reduce error you step the opposite way — that is Gradient Descent.

Why the topic needs it. Backprop's entire job is to compute this gradient cheaply. Gradient Descent then consumes it.

4. The chain rule (why slopes multiply)

The picture: two boxes in a line. A tiny push $dx$ enters, gets scaled by the first box's slope, then scaled again by the second box's slope. Two multiplications, in order.

Fan-in case (the sum). If $x$ affects $L$ through several routes $y_1,\dots ,y_k$, each route contributes its own multiplied slope and we add them: $\frac{\partial L}{\partial x}={\sum }_{i=1}^k\frac{\partial L}{\partial y_i}\frac{\partial y_i}{\partial x}.$

Why the sum? A change in $x$ ripples down every wire it feeds; the total effect is all those effects piled together. This is the multivariable chain rule and it is the reason the parent warns "at a fan-in node, sum over all paths."

Why the topic needs it. Backprop = chain rule applied backward through a computation graph, reusing cached numbers.

5. Vectors, matrices, and the transpose $\top$

The picture: a matrix is a wiring board — row $i$, column $j$ entry $W_{ij}$ is the strength of the wire from input $j$ to output $i$.

Why the topic needs it. The forward pass sends signals through $W$. Sending the error backward uses $W^{\top }$ — because the slope of the linear map $Wa$ with respect to $a$ is exactly $W^{\top }$. The parent's most-feared mistake ("backprop needs the inverse") is fixed by this: it is a transpose, not an inverse. Reversing wires ≠ undoing them.

6. The elementwise product $\odot$

The picture: two parallel columns of numbers; multiply each pair on the same row.

Why the topic needs it. Each unit scales its own incoming error by its own slope ${\sigma }^{\prime }(z)$. Same-slot scaling = $\odot$. That is the $\odot {\sigma }^{\prime }(z)$ in the parent's boxed backprop equation.

7. The activation $\sigma$ and its slope ${\sigma }^{\prime }$

Why the topic needs it. If ${\sigma }^{\prime }(z)\approx 0$ (a "saturated"/sleepy unit), almost no error passes back — this is the seed of Vanishing Gradients. If we forget the ${\sigma }^{\prime }$ factor entirely, gradients through nonlinear units are simply wrong.

8. The rest of the parent's alphabet (quick glossary)

Why $\delta$ (delta) matters most. It is the reusable quantity: compute it once per layer, and both that layer's weight-gradient and the next-layer-down's $\delta$ fall out of it. Reusing $\delta$ is why backprop is cheap — the Automatic Differentiation idea in disguise.

Prerequisite map

Each foundation on the left feeds the backprop node on the right; backprop then feeds the training loop.

Equipment checklist

Self-test: can you answer each without peeking?

Connections

• 5.6.08 Backpropagation — chain rule, gradient computation (Hinglish) — the parent, in Hinglish.
• Chain Rule — the multiplication-of-slopes engine built in Section 4.
• Gradient Descent — consumes the gradient we learned to name in Section 3.
• Computational Graphs — the wired-boxes picture from Sections 0 and 8.
• Activation Functions — the bendy box $\sigma$ of Section 7.
• Vanishing Gradients — what happens when ${\sigma }^{\prime }(z)\approx 0$.
• Automatic Differentiation — the reuse-$\delta$ idea generalised by machines.
• Neural Network Surrogate Models (CFD) — where all this pays off in aerospace.