Foundations — Backpropagation algorithm derivation

This page assumes nothing. Before you can read the parent derivation, you must be fluent in the little alphabet it speaks. We build every letter here, in order, each one leaning on the one before it.

0 · What a neural network physically is

Picture a row of light bulbs feeding wires into another row of bulbs, feeding into another row, ending in one final bulb whose brightness is the network's "answer."

• Each circle is a neuron — a tiny number-holder (its brightness).
• Each arrow is a weight — a dial that says "how strongly does this bulb influence that one."
• Numbers flow left → right. This left-to-right flow is the forward pass.

Everything below is just names for the parts of this picture.

1 · The layer index $l$ — which row are we in?

Picture: the vertical columns in the figure above, numbered left to right. Why the topic needs it: every quantity ($z$, $a$, $W$, $b$, $\delta$) lives in a specific row, so we tag it with a superscript ${}^{(l)}$ to say which row. $W^{(2)}$ means "the dials feeding into row 2."

2 · The activation $a$ — how bright is a bulb?

Picture: $a_j^{(l)}$ is the brightness of the $j$-th bulb in row $l$.

Two anchors the parent uses immediately:

• $a^{(0)}=x$ — the input row's activations are the input data $x$.
• $a^{(L)}=\hat{y}$ — the last row's activation is the network's prediction, written $\hat{y}$ ("y-hat", the guessed $y$).

3 · Weights $W$ and biases $b$ — the dials

Picture: each arrow between two bulbs carries one $W_{ji}$; each destination bulb also has a little "+$b$" tap. Why the topic needs it: these are the knobs training adjusts. The entire goal of backprop is to find $\partial L/\partial W$ and $\partial L/\partial b$ — how to turn each dial.

4 · The pre-activation $z$ and the activation function $\sigma$

Each bulb does its job in two steps.

Step A — mix the inputs (linear).

Step B — squash it (nonlinear).

$a^{(l)}=\sigma (z^{(l)})$

Why two steps and not one? Without the curved $\sigma$, stacking rows would collapse into one big linear map (a line of lines is still a line). The bend is what makes a deep network more powerful than a shallow one. (More in Activation functions and their derivatives.)

5 · Why vectors and matrices at all? The dot product

Row $j$'s raw score is $z_j^{(l)}={\sum }_i{W}_{ji}^{(l)}{a}_i^{(l-1)}+b_j^{(l)}$ — "walk across all sources $i$, multiply dial × brightness, add them up." That summed-products pattern is the dot product, and stacking it for every destination $j$ is matrix multiplication.

6 · The loss $L$ — how wrong are we?

Picture: a ruler measuring the distance between the final bulb's brightness and where we wanted it. Example used by the parent: MSE, $L=\frac{1}{2}(a^{(L)}-y{)}^2$. (See Loss functions (MSE, cross-entropy).) Why the topic needs it: $L$ is the mountain we descend. Every derivative in backprop is "how does $L$ change if I wiggle this?"

7 · The derivative and the gradient — the slope of the mountain

Picture: stand on a hillside (the loss surface); the gradient is the arrow pointing straight uphill. To lower loss we step the opposite way — that is Gradient descent.

8 · The chain rule — the engine of the whole topic

Suppose $L$ depends on $z$, which depends on $w$. How does $L$ respond to $w$?

When $w$ influences $L$ through several paths (because a neuron fans out to many neurons in the next row), you add up the contributions from every path:

$\frac{\partial L}{\partial w}={\sum }_k\frac{\partial L}{\partial z_k}\cdot \frac{\partial z_k}{\partial w}$

Why the topic needs it: the entire parent derivation is repeated chain-rule. The sum-over-paths version is exactly the ${\sum }_k$ that produces BP2's transpose.

9 · The Hadamard product $\odot$

Picture: two stacks of numbers side by side; multiply each pair, keep the stack the same height. Why the topic needs it: in BP1/BP2, the incoming blame vector must be "gated" by each neuron's own local slope ${\sigma }^{\prime }(z)$ — a per-neuron multiply, i.e. $\odot$, not a matrix product.

10 · The error signal $\delta$ — the star of backprop

Picture: the "blame packet" the parent's Feynman story passes backward from person to person. Why the topic needs it: $\delta$ is the reusable cache. Compute it once per row (back to front) and every weight/bias gradient falls out cheaply — this is the dynamic-programming saving. Chained naively you'd recompute shared paths (linked to Vanishing and exploding gradients behaviour too).

Prerequisite map

Each block must be solid before the next makes sense; the final block is the parent topic.

Equipment checklist