Visual walkthrough — Implementing models from scratch

We will need the ideas of a neuron layer, the chain rule, matrix multiplication, and backpropagation — but we will re-derive each piece as we go.

---

Step 1 — One neuron, one number

WHAT. Forget matrices. Start with the smallest possible piece: one input number $x$, one weight $w$, one bias $b$, giving one output number $z=\underset{\text{how strongly }x\text{ counts}}{\underbrace{w}}\cdot \underset{\text{the input}}{\underbrace{x}}+\underset{\text{a constant shift}}{\underbrace{b}}.$

WHY. Every big formula in the parent note is this line copied many times. If we understand how the loss $L$ (a single number measuring "how wrong we are") depends on $w$ and $b$ here, the matrix version is just bookkeeping.

PICTURE. Below, the input $x$ enters on the left, gets multiplied by $w$, has $b$ added, and out comes $z$. Think of $w$ as the slope of a straight line and $b$ as where the line crosses zero.

---

Step 2 — The single question backprop answers

WHAT. We want to nudge $w$ and $b$ to make $L$ smaller. To nudge intelligently we must know: "If I increase $w$ by a tiny amount, does $L$ go up or down, and how fast?" That number is the derivative $\frac{\partial L}{\partial w}$.

WHY a derivative and not something else? Because "how fast does $L$ change as $w$ changes" is exactly the definition of a derivative — the slope of the $L$-vs-$w$ curve at our current spot. No other tool answers "which way is downhill" locally.

PICTURE. Plot $L$ against $w$. At our current $w$, draw the tangent line. Its slope is $\partial L/\partial w$. If the slope is positive, increasing $w$ raises $L$ — so we must go the other way. That sign is everything.

---

Step 3 — The activation, and how the error $\delta$ is actually born

WHAT. Between the neuron's raw output $z$ and the loss $L$ sits one more stage we must name: the activation function $\sigma$. It takes $z$ and bends it into $a=\sigma (z)$, the neuron's final "firing strength". A common choice is $\sigma (z)=\text{ReLU}(z)=\max (0,z)$. So the real chain is three links long: $w⟶z=wx+b⟶a=\sigma (z)⟶L.$

WHY do we need $\sigma$ at all? Without it, stacking layers collapses to one big linear map (the parent note's argument). $\sigma$ injects the bend that lets networks learn curves. And because $L$ sees $z$ only through $a$, the error signal at $z$ must pass through $\sigma$'s slope first.

PICTURE. Three boxes in a row: $z\to \sigma \to a\to L$. The error travels right-to-left; at the $\sigma$ box it gets multiplied by ${\sigma }^{\prime }(z)$, the local steepness of the bend.

Now compute $\delta$ by the chain rule, one link at a time. Suppose the next stage hands us $\frac{\partial L}{\partial a}$ (how loss reacts to this neuron's output). Then: $\delta =\frac{\partial L}{\partial z}=\underset{\text{given from above}}{\underbrace{\frac{\partial L}{\partial a}}}\cdot \underset{={\sigma }^{\prime }(z)}{\underbrace{\frac{\partial a}{\partial z}}}=\frac{\partial L}{\partial a}{\sigma }^{\prime }(z).$

WHY multiply by ${\sigma }^{\prime }(z)$? $z$ only reaches $L$ through $a$. If the bend is flat there (${\sigma }^{\prime }(z)=0$), wiggling $z$ does nothing to $a$, hence nothing to $L$ — so $\delta =0$. This is the missing piece: $\delta$ is not handed down from heaven; it is $\partial L/\partial a$ passed through the activation's slope.

---

Step 4 — Chain rule: from $\delta$ down to $w$ and $b$

WHAT. Now that $\delta =\partial L/\partial z$ is computed, $w$ reaches $L$ only through $z$, so one more chain-rule link finishes the job: $\frac{\partial L}{\partial w}=\underset{=\delta }{\underbrace{\frac{\partial L}{\partial z}}}\times \underset{\text{how }z\text{ reacts to }w}{\underbrace{\frac{\partial z}{\partial w}}}.$

WHY multiply? If wiggling $w$ moves $z$ by 3 units, and each unit of $z$ moves $L$ by $\delta$ units, then $w$ moves $L$ by $3\delta$. Chained effects multiply — this is the core reason the whole algorithm is called backpropagation: we carry one number ($\delta$) backward and keep multiplying.

PICTURE. Two gears in a chain. Turn the $w$-gear; it drives the $z$-gear; that (through $\delta$) drives $L$. Total gear ratio = product of the two.

From $z=wx+b$, differentiate treating everything else as constant: $\frac{\partial z}{\partial w}=x\frac{\partial z}{\partial b}=1.$

WHY these values? Bump $w$ by a tiny $\epsilon$: $z$ changes by $x\epsilon$, so the rate is $x$. Bump $b$ by $\epsilon$: $z$ changes by exactly $\epsilon$, so the rate is $1$.

Plug in: $\boxed{\frac{\partial L}{\partial w}=\delta \cdot x\frac{\partial L}{\partial b}=\delta \cdot 1=\delta }$

---

Step 5 — Widen it: many inputs, many outputs (the matrix appears)

WHAT. Real layers have many inputs $x_1,\dots ,x_n$ and many outputs $z_1,\dots ,z_m$. Each output $z_i$ is Step-1's line, but summing over all inputs: $z_i={\sum }_{j=1}^n{W}_{ij}{x}_j+b_i.$ Here $W_{ij}$ is the weight from input $j$ to output $i$ — a whole grid of numbers, the matrix $\mathbf{W}$. The bias becomes a column vector $\mathbf{b}\in {\mathbb{R}}^{m\times 1}$ — one bias per output.

WHY a grid? Every (output, input) pair needs its own knob, and there are $m\times n$ such pairs. Stacking them makes $\mathbf{z}=\mathbf{Wx}+\mathbf{b}$ — the compact form.

PICTURE. A bipartite graph: $n$ input dots on the left, $m$ output dots on the right, one arrow per weight $W_{ij}$. The grid of arrows is the matrix.

Now Step 4's rule, applied to entry $W_{ij}$, becomes $\frac{\partial L}{\partial W_{ij}}={\delta }_i{x}_j$ (the error on output $i$ times the input $j$ that fed it). Writing every such product at once is exactly an outer product: $\frac{\partial L}{\partial \mathbf{W}}=\mathbf{\delta }{\mathbf{x}}^T.$

---

Step 6 — Sending the error further back: the other transpose

WHAT. To keep going backward we need $\partial L/\partial x_j$ — how the loss depends on each input, so the previous layer can use it (through its own ${\sigma }^{\prime }$) as its $\delta$. Input $x_j$ feeds every output $z_i$ (look again at the fan of arrows), so its influences add up: $\frac{\partial L}{\partial x_j}={\sum }_{i=1}^m\underset{\text{error at }{z}_i}{\underbrace{{\delta }_i}}\underset{\text{arrow }j\to i}{\underbrace{W_{ij}}}.$

WHY sum over $i$? One input pushes on many outputs; the chain rule adds all those paths. That sum ${\sum }_i{W}_{ij}{\delta }_i$ is exactly row $j$ of ${\mathbf{W}}^T$ dotted with $\mathbf{\delta }$: $\frac{\partial L}{\partial \mathbf{x}}={\mathbf{W}}^T\mathbf{\delta }.$

PICTURE. Reverse every arrow. Forward, $\mathbf{W}$ sent $\mathbf{x}\to \mathbf{z}$; backward, the same weights read the other direction (${\mathbf{W}}^T$) carry the error $\mathbf{z}\to \mathbf{x}$.

---

Step 7 — Batches, and where the $1/B$ really lives

WHAT. We rarely feed one sample. Stack $B$ samples as columns: $\mathbf{X}\in {\mathbb{R}}^{n\times B}$, $\mathbf{\delta }\in {\mathbb{R}}^{m\times B}$. First we must be honest about what $L$ means over a batch. We define the batch loss as the average of the per-sample losses, $L=\frac{1}{B}{\sum }_{s=1}^B{L}^{(s)},$ where $L^{(s)}$ is the loss on sample $s$. Because the $\frac{1}{B}$ lives in the definition of $L$, it automatically appears in every gradient — we do not sprinkle it in by hand. With that convention:

\frac{\partial L}{\partial \mathbf{b}} = \frac{1}{B}\sum_{\text{columns}}\boldsymbol{\delta}.$$ **WHY sum the bias, but matrix-multiply the weight?** The outer product $\boldsymbol\delta\mathbf X^T$ *already* sums over the batch (the shared column index does that). The bias has no input to multiply, so every sample contributes its raw $\delta_i$ and we sum the $\delta$ columns directly. Both then carry the same $\tfrac1B$ inherited from $L$'s definition — nothing extra. **PICTURE.** $B$ transparent copies of the layer stacked in depth; the bias gradient is the vertical column-sum of all their error signals, then scaled by $1/B$. ![[deepdives/dd-ai-ml-6.5.02-d2-s07.png]] > [!mistake] Forgetting `keepdims` > Our shape convention (fixed in Step 5) is $\mathbf b\in\mathbb R^{m\times 1}$ — a **column**. > But `np.sum(dZ, axis=1)` collapses to shape $(m,)$ — a flat 1-D array with **no** second axis. > Adding a $(m,)$ update to an $(m,1)$ bias makes NumPy broadcast into an $(m,m)$ grid silently, and > you update the wrong thing. `keepdims=True` preserves the $(m,1)$ column so shapes match. --- ## Step 8 — The degenerate cases (never skip these) **WHAT & WHY.** Every real training run hits edges. Here is what each formula *does* at the edge: | Situation | What happens | Why | |---|---|---| | $x_j = 0$ (dead input) | $\partial L/\partial W_{ij}=\delta_i\cdot 0=0$ | that weight saw no signal, so it gets **no** update — correct. | | $\partial L/\partial a = 0$ (output already right) | $\delta_i=0$, its whole weight-gradient row is $0$ | leave that output alone. | | ReLU with $z_i<0$ | $\sigma'(z_i)=0$, so $\delta_i=\tfrac{\partial L}{\partial a}\cdot 0=0$ | the neuron was "off"; changing $z$ wouldn't have changed $a$. | | $B=1$ (single sample) | the $\tfrac1B$ is just $1$; formulas unchanged | batch is a generalisation, not a new rule. | | all inputs equal, $\alpha$ too big | update overshoots, $L$ rises | Step 2's tangent is only local; step must be small. | **PICTURE.** The ReLU gate: for $z<0$ the "valve" is shut and the backward error $\delta$ is zeroed by $\sigma'=0$; for $z>0$ it passes untouched ($\sigma'=1$). ![[deepdives/dd-ai-ml-6.5.02-d2-s08.png]] > [!intuition] "Dead neuron" made visible > A neuron with $z\le 0$ outputs $0$ *and* blocks its own gradient (since $\sigma'(z)=0$ kills > $\delta$). If it stays negative for all data it never learns again — the famous **dying ReLU**. > The picture is a permanently shut valve. --- ## Step 9 — The update: what we finally do with the gradients **WHAT.** We now have $\partial L/\partial\mathbf W$ and $\partial L/\partial\mathbf b$. Collect **all** the tunable numbers of the layer — every entry of $\mathbf W$ and $\mathbf b$ — into one big bundle we call $\boldsymbol\theta$ (theta), the ==parameters==. Its matching bundle of gradients is written $\nabla_{\boldsymbol\theta}L$ (read "grad L"): same shape as $\boldsymbol\theta$, holding $\partial L/\partial\theta_k$ in each slot. Then take one step *downhill* (Step 2's rule): $$\boldsymbol\theta \;\leftarrow\; \boldsymbol\theta \;-\; \underbrace{\alpha}_{\text{step size}}\;\underbrace{\nabla_{\boldsymbol\theta}L}_{\text{uphill direction}}.$$ **WHY these symbols?** - $\boldsymbol\theta$ ::: just a name for "all weights and biases piled together" so we can write one update instead of two. - $\nabla_{\boldsymbol\theta}L$ ::: the pile of slopes $\partial L/\partial\theta_k$ — the direction of steepest *increase* of $L$. - $\alpha$ ::: the ==learning rate==, a small positive number choosing how far we step. Too big overshoots (Step 8's last row); too small crawls. The **minus sign** turns "steepest uphill" into "step downhill", exactly Step 2's picture generalised to many parameters at once. This is [[4.2.02-Gradient-Descent|gradient descent]]; smarter variants live in [[6.3.04-Optimization-Algorithms|optimization algorithms]]. **PICTURE.** One figure, the full round-trip: forward $\mathbf{x}\xrightarrow{\mathbf W}\mathbf z\xrightarrow{\sigma}\mathbf a\to L$, then backward $\delta=\tfrac{\partial L}{\partial a}\sigma'(z)$ flowing right-to-left, spawning $\partial L/\partial\mathbf W$ and $\partial L/\partial\mathbf b$, then the step $\boldsymbol\theta \leftarrow \boldsymbol\theta - \alpha\nabla_{\boldsymbol\theta}L$. ![[deepdives/dd-ai-ml-6.5.02-d2-s09.png]] > [!recall]- Feynman retelling — say it like a story > A number $x$ walks into a machine, gets multiplied by a weight $w$ and shifted by a bias $b$ — out > pops $z$. An activation $\sigma$ then bends $z$ into $a$, and only through $a$ does the loss $L$ > see the neuron. To learn, the far end hands back $\partial L/\partial a$; we pass it through the > bend's slope $\sigma'(z)$ to get the error at $z$, called $\delta = \partial L/\partial a\cdot\sigma'(z)$. > To update a weight we ask the chain rule again: the answer is $\delta$ (the error) times $x$ (the > input that weight saw) — big input, big update; zero input, zero update. Stack all those products > and you get $\boldsymbol\delta\mathbf X^T$; the transpose is just the shape that makes > column-times-row into the weight grid. The bias (a column $\mathbf b\in\mathbb R^{m\times 1}$) has > no input to multiply, so it simply **sums** the errors across the batch. To keep the story going > into the previous layer, we run the *same* weights in reverse — that is $\mathbf W^T\boldsymbol\delta$. > Finally we pile all weights and biases into $\boldsymbol\theta$ and step each a little bit > *downhill* — opposite $\nabla_{\boldsymbol\theta}L$, by an amount $\alpha$. Because $L$ is defined > as the *average* over the batch, the $1/B$ is already baked in. Repeat millions of times and the > machine learns. No magic — pass the error through the bend, multiply the gear ratios, and always > walk downhill. > [!recall]- Quick self-test > How is the error $\delta$ at $z$ computed from above? ::: $\delta = \dfrac{\partial L}{\partial a}\cdot\sigma'(z)$ — the incoming gradient passed through the activation's slope. > Why is the weight gradient an outer product and not element-wise? ::: Because each weight $W_{ij}$ needs its own $\delta_i x_j$, and column $\boldsymbol\delta$ times row $\mathbf x^T$ produces exactly that grid. > Why does the backward pass use $\mathbf W^T$? ::: Forward, outputs sum over inputs (rows); backward, inputs sum over outputs (columns = rows of $\mathbf W^T$). > Where does the $1/B$ come from? ::: From defining the batch loss $L$ as the average of per-sample losses; it is inherited by every gradient automatically. > What are $\boldsymbol\theta$, $\alpha$, and $\nabla_{\boldsymbol\theta}L$ in the update? ::: All parameters piled together, the learning-rate step size, and the pile of slopes $\partial L/\partial\theta_k$.