Implementing models from scratch: Worked examples

This page pushes on the machinery from the parent note: the forward pass ( $Z = WX + b$ ), the backward pass (chain rule → gradients), and the update ( $θ \leftarrow θ - α \nabla L$ ). Instead of new theory, we drill every case a single dense layer can throw at you: positive and negative activations, dead neurons, zero gradients, a full batch, and the exam-style "predict the shape" twist.

Before the first line, three plain-word reminders so nobody is lost:

One more operation earns its keep on every single forward pass, so we define it now:

Everything below is a single layer computing $Z = WX + b$ , sometimes followed by ReLU. We will hand-crank the arithmetic — no PyTorch.

The scenario matrix

Here is every distinct kind of situation this topic can hand you. Each worked example below is tagged with the cell it fills.

#	Case class	What makes it different	Example
A	Single sample, all positive	baseline forward, no sign trouble	Ex 1
B	Single sample, some negatives	one negative pre-activation feeds ReLU	Ex 2
C	Zero / degenerate input	input is the zero vector	Ex 3
D	Dead neuron (zero gradient)	ReLU gate blocks the gradient	Ex 4
E	Full mini-batch forward	broadcasting bias across columns	Ex 5
F	Backward: weight gradient	$δ X^{T}$ , sum over batch	Ex 6
G	Backward: bias gradient	sum across batch dimension	Ex 6
H	One SGD step (real-world word problem)	numbers actually move downhill	Ex 7
I	Limiting value (learning rate too big)	update overshoots, loss rises	Ex 8
J	Exam twist: shape-only reasoning	no numbers, just dimensions	Ex 9
K	All neurons dead	every pre-activation $\leq 0$	Ex 10

Recall Which cells involve NO calculus at all?

Cases A, B, C, E, J, K ::: they are pure forward-pass or shape bookkeeping — matrix multiply plus a max, or just counting dimensions.

Setup used by all numeric examples

To keep arithmetic checkable by hand we fix a tiny layer: 2 inputs → 2 outputs.

$W = [10 - 2 3], b = [1 - 1]$

Read $W$ row by row: neuron 1 computes $1 \cdot x_{1} - 2 \cdot x_{2}$ , neuron 2 computes $0 \cdot x_{1} + 3 \cdot x_{2}$ . The bias then shifts each: $+ 1$ for neuron 1, $- 1$ for neuron 2.

The wiring diagram below makes this concrete: two purple input nodes on the left send weighted wires (orange for neuron 1, teal for neuron 2) into the two output neurons. Notice the dotted grey wire labelled $w = 0$ — that is the $0$ in the first column of row 2, meaning neuron 2 completely ignores $x_{1}$ . Trace the orange wires and you can read off neuron 1's formula; trace the teal ones for neuron 2. Keep this picture in mind for every forward pass below.

Figure — Implementing models from scratch

Figure 1 — The 2→2 dense layer. Purple = inputs, orange wires build neuron 1, teal wires build neuron 2, the dotted $w = 0$ wire is a weight that carries nothing.

Case A — single sample, all positive

Worked example Ex 1 — the clean baseline (cell A)

Input $x = [21]$ . Compute $z = Wx + b$ , then $a = ReLU (z)$ .

Forecast: guess the two numbers in $z$ before reading on.

Row 1 of $W$ times $x$ : the dot product $1 \cdot 2 + (- 2) \cdot 1 = 0$ . Why this step? Each output number is one dot product (defined above) of a weight-row with the input — that is literally what " $Wx$ " means, row against column.
Row 2: $0 \cdot 2 + 3 \cdot 1 = 3$ . So $Wx = [03]$ .
Add bias: $[03] + [1 - 1] = [12]$ . Why this step? The bias translates the output — it lets a neuron fire even when the weighted sum is zero.
ReLU: $max (0, 1) = 1$ , $max (0, 2) = 2 \Rightarrow a = [12]$ . Why this step? ReLU keeps positives as-is; both are positive so nothing changes.

Verify: both entries are $\geq 0$ and unchanged by ReLU, consistent with all pre-activations being positive. $z = (1, 2)$ , $a = (1, 2)$ . ✓

Case B — single sample, some negatives

Case C — zero / degenerate input

Worked example Ex 3 — the zero vector (cell C)

Input $x = [00]$ . What does the layer output?

Forecast: does zero input mean zero output? Careful — there's a bias.

$Wx$ : every dot product with the zero column is $0$ , so $Wx = [00]$ . Why? Multiplying anything by zero gives zero; the weights never get a chance to act.
Add bias: $z = [00] + [1 - 1] = [1 - 1]$ . Why this matters? The output is not zero — the bias is exactly what a neuron says when it sees nothing.
ReLU: $max (0, 1) = 1$ , $max (0, - 1) = 0 \Rightarrow a = [10]$ .

Verify: with zero input, output equals $ReLU (b)$ . Indeed $ReLU (1, - 1) = (1, 0)$ . ✓

Case D — the dead neuron (zero gradient)

To do backprop we need a loss. Use the simplest: suppose the next layer sent back the gradient of the loss with respect to the activations, $\frac{\partial L}{\partial a} = [2 - 1] (call it g) .$ ReLU then decides which of these gradients survive. Recall (from the parent, and 6.2.03-Backpropagation-Algorithm): $\frac{\partial L}{\partial z _{i}} = g_{i} \cdot [z_{i} > 0]$ where $[z_{i} > 0]$ is $1$ if true, $0$ if false — the ReLU gate.

Common mistake What happens exactly at

$z_{i} = 0$ ? Look at ReLU near zero: to the left the slope is $0$ , to the right it is $1$ . At the single point $z_{i} = 0$ the function has a corner, so it has no unique slope — the true mathematical derivative does not exist there. In practice we simply pick one value, and the universal convention (used by every framework) is to send the gate to $0$ at $z_{i} = 0$ , which is why the rule is written with the strict inequality $[z_{i} > 0]$ (so $z_{i} = 0$ falls into the "off" branch). Why choose $0$ and not $1$ ? Because a neuron sitting exactly at $0$ produced output $0$ in the forward pass — it contributed nothing — so it is honest to let it also contribute no gradient. The choice is harmless anyway: hitting $z_{i}$ exactly $0$ is a measure-zero fluke, and either choice ( $0$ or $1$ ) gives a valid subgradient. This is exactly why Ex 2/Ex 4 treat $z \leq 0$ as gate $= 0$ .

Before we compute, one piece of notation we will lean on:

The figure below shows why a gate of $0$ or $1$ is exactly the right thing to $⊙$ with the incoming gradient. The orange curve is ReLU itself; the dashed teal curve is its slope — flat at $0$ for negative inputs, flat at $1$ for positive inputs, with the corner at the origin marked. The two purple markers sit at our example pre-activations $z_{1} = - 4$ (slope $0$ , gradient blocked) and $z_{2} = 5$ (slope $1$ , gradient passes). Read the dashed line under each marker to see whether that neuron's gradient survives.

Figure 2 — ReLU (orange) passes positives unchanged; its slope (dashed teal) is the gate you multiply the backward gradient by. At $z_{1} = - 4$ the gate is $0$ (gradient dies); at $z_{2} = 5$ the gate is $1$ (gradient flows); at the origin corner the slope is undefined and we choose $0$ .

Worked example Ex 4 — gradient gets gated (cell D)

Use the pre-activations from Ex 2: $z = [- 4 5]$ , incoming $g = [2 - 1]$ . Find $δ = \frac{\partial L}{\partial z}$ .

Forecast: one of these two gradient numbers will be killed. Which?

Build the gate mask: $z_{1} = - 4 \leq 0 \Rightarrow$ gate $= 0$ ; $z_{2} = 5 > 0 \Rightarrow$ gate $= 1$ . Mask $= [01]$ . Why? ReLU's slope is $0$ where its input was negative and $1$ where positive — the mask is the derivative (with the $z = 0$ corner sent to $0$ as discussed).
Multiply element-wise: $δ = g ⊙ mask = [2 \cdot 0 - 1 \cdot 1] = [0 - 1]$ . Why $⊙$ and not matrix multiply? Each neuron's gate touches only its own gradient — no mixing across neurons — so it is pointwise, exactly what $⊙$ means.

Verify: neuron 1 output $0$ in the forward pass, so nudging its weights can't change $L$ — its gradient must be $0$ , and it is. $δ = (0, - 1)$ . ✓ This is the famous dead-neuron effect.

Case E — a full mini-batch forward

Now feed three samples at once, stacked as columns of $X$ : $X = [21 - 1 2 00] (these are Ex 1, Ex 2, Ex 3 side by side) .$

Worked example Ex 5 — broadcasting the bias (cell E)

Compute $Z = WX + b$ for the whole batch.

Forecast: you already did the three columns individually — can you assemble them?

$WX$ column by column: each column is an independent matrix–vector product. From Ex 1/2/3: columns of $WX$ are $(0, 3)$ , $(- 5, 6)$ , $(0, 0)$ . Why columns? A batch matmul is just the same layer applied to each sample — nothing couples the samples.
Add the bias to every column (broadcasting): $b = (1, - 1)$ is added to all three. Why broadcast? The layer's bias is shared by all samples — the same shift applies everywhere. $Z = [0 + 1 3 - 1 - 5 + 1 6 - 1 0 + 1 0 - 1] = [12 - 4 5 1 - 1]$

Verify: column 1 is $(1, 2)$ = Ex 1's $z$ ✓, column 2 is $(- 4, 5)$ = Ex 2 ✓, column 3 is $(1, - 1)$ = Ex 3 ✓. Shape is $2 \times 3$ = (neurons × samples). ✓

Cases F & G — weight and bias gradients over the batch

Suppose the upstream gradient at this layer's pre-activations is $δ = [1 - 1 0320] (2 neurons \times 3 samples) .$ The parent gives $\frac{\partial L}{\partial W} = δ X^{T}$ and $\frac{\partial L}{\partial b} = \sum_{batch} δ$ .

Worked example Ex 6 — hand-cranking

$δ X^{T}$ and the bias sum (cells F & G) Forecast: $\frac{\partial L}{\partial W}$ must come out the same shape as $W$ , i.e. $2 \times 2$ . $\frac{\partial L}{\partial b}$ must be $2 \times 1$ .

Transpose $X$ : $X^{T} = 2 - 1 0 120$ (samples become rows). Why transpose? We need to line up "for each neuron's error, multiply by the input feature that caused it," and the transpose makes the inner dimensions ( $3$ samples) match so they sum away.
Multiply $δ (2 \times 3) \cdot X^{T} (3 \times 2)$ : the shared $3$ collapses, summing each sample's contribution.
- Row 1: $[1, 0, 2] \cdot$ columns of $X^{T}$ : $(1 \cdot 2 + 0 \cdot (- 1) + 2 \cdot 0, 1 \cdot 1 + 0 \cdot 2 + 2 \cdot 0) = (2, 1)$ .
- Row 2: $[- 1, 3, 0]$ : $(- 1 \cdot 2 + 3 \cdot (- 1) + 0, - 1 \cdot 1 + 3 \cdot 2 + 0) = (- 5, 5)$ . $\frac{\partial L}{\partial W} = [2 - 5 15]$
Bias = sum each row of $δ$ across samples: $(1 + 0 + 2, - 1 + 3 + 0) = (3, 2)$ . Why sum? The bias is reused for all samples, so its total sensitivity is the running total of the per-sample sensitivities. See 3.1.04-Chain-Rule. $\frac{\partial L}{\partial b} = [32]$

Verify: $\frac{\partial L}{\partial W}$ is $2 \times 2$ (matches $W$ ) ✓, $\frac{\partial L}{\partial b}$ is $2 \times 1$ (matches $b$ ) ✓. Shapes are the acid test that the transposes are correct.

Case H — one real gradient-descent step (word problem)

Worked example Ex 7 — a thermostat learns (cell H)

Story: A single weight $w$ predicts room temperature. Current $w = 4.0$ . On this batch the loss gradient is $\frac{\partial L}{\partial w} = + 6.0$ (loss rises when $w$ rises). Learning rate $α = 0.1$ . Take one SGD step (see 6.3.04-Optimization-Algorithms).

Forecast: the gradient is positive — will $w$ go up or down?

Apply the rule $w \leftarrow w - α \frac{\partial L}{\partial w}$ : $w = 4.0 - 0.1 \cdot 6.0 = 4.0 - 0.6 = 3.4$ . Why subtract? The gradient points in the direction that increases the loss (a positive slope means "loss climbs as $w$ climbs"). To decrease the loss we must walk the opposite way — so we subtract the gradient rather than add it. That single minus sign is the whole idea of gradient descent: always step downhill on the loss surface, and downhill is the negative-gradient direction.
Check the loss actually dropped (local, first-order): the predicted change is $Δ L \approx \frac{\partial L}{\partial w} \cdot Δ w = 6.0 \cdot (- 0.6) = - 3.6 < 0$ . Why this check? The parent's Taylor argument says loss should fall by about $α ∥\nabla L ∥^{2} = 0.1 \cdot 36 = 3.6$ — a negative $Δ L$ confirms we chose the right direction and a small enough step.

Verify: new $w = 3.4$ (moved down, correct direction) and predicted $Δ L = - 3.6$ matches $- α ∥\nabla L ∥^{2}$ . ✓

Case I — limiting value: learning rate too big

Worked example Ex 8 — overshoot (cell I)

Same subtraction rule as Ex 7 but with a giant $α = 1.0$ , and suppose the true loss is the bowl $L (w) = (w - 1)^{2}$ , so $\frac{\partial L}{\partial w} = 2 (w - 1)$ . Start $w = 4.0$ .

Forecast: the minimum is at $w = 1$ . Will one big step land near it, or fly past?

Gradient at $w = 4$ : $2 (4 - 1) = 6$ . Why? Slope of the bowl at $4$ is steep and positive.
Step: $w = 4 - 1.0 \cdot 6 = - 2$ . Why did we overshoot? We wanted to reach $1$ , needed to drop by $3$ , but $α \cdot 6 = 6$ dropped us by $6$ — twice too far.
Loss before vs after: $L (4) = 9$ , $L (- 2) = (- 3)^{2} = 9$ . Why equal? We landed the exact mirror distance on the other wall of the bowl.

Verify: loss did not decrease ( $9 \to 9$ ); with $α$ this large gradient descent stalls or diverges. Contrast Ex 7 where a small $α$ made progress. $w$ went $4 \to - 2$ , $L$ stayed $9$ . ✓

Case J — exam twist: reason with shapes only

Worked example Ex 9 — no numbers, just dimensions (cell J)

Statement: A layer maps $R^{7} \to R^{3}$ and processes a batch of $16$ samples with the columns-as-samples convention. Give the shapes of $W$ , $X$ , $Z$ , $b$ , and both gradients $\frac{\partial L}{\partial W}$ and $\frac{\partial L}{\partial X}$ .

Forecast: write the six shapes down before reading — a smart exam trick is that every gradient copies the shape of the thing it differentiates.

$W$ is (out × in) $= 3 \times 7$ . Why? Each of the $3$ output neurons owns a full row of $7$ weights, one per input feature.
$X$ is (in × batch) $= 7 \times 16$ , and $Z = WX$ is $(3 \times 7) (7 \times 16) = 3 \times 16$ . Why? The shared inner dimension $7$ collapses in the matrix product, leaving (neurons × samples).
$b$ is $3 \times 1$ , broadcast across all $16$ columns. Why? One bias per output neuron, reused for every sample.
$\frac{\partial L}{\partial W} = δ X^{T}$ is $(3 \times 16) (16 \times 7) = 3 \times 7$ — identical to $W$ . Why must it match? During the update you compute $W - α \partial_{W} L$ ; subtraction demands equal shapes.
$\frac{\partial L}{\partial X} = W^{T} δ$ is $(7 \times 3) (3 \times 16) = 7 \times 16$ — identical to $X$ , ready to hand back to the previous layer.

Verify: each gradient matches the shape of the quantity it differentiates. $W : 3 \times 7$ , $X : 7 \times 16$ , $Z : 3 \times 16$ , $b : 3 \times 1$ , $\partial_{W} L : 3 \times 7$ , $\partial_{X} L : 7 \times 16$ . ✓

Case K — every neuron dead

Worked example Ex 10 — a fully silent layer (cell K)

Same $W, b$ . Input $x = [32]$ . Compute $z$ , $a$ , and then the pre-activation gradient $δ$ if the incoming gradient is $g = [57]$ .

Forecast: with these inputs, could both neurons go dark at once?

Row 1: $1 \cdot 3 + (- 2) \cdot 2 = 3 - 4 = - 1$ ; add bias $+ 1 \Rightarrow 0$ . Why watch this one? It lands exactly on the corner $z_{1} = 0$ — the case we discussed above.
Row 2: $0 \cdot 3 + 3 \cdot 2 = 6$ ; add bias $- 1 \Rightarrow 5$ . Wait — that is positive. Adjust the input to make the story honest: retry with $x = [3 - 1]$ . Row 1: $3 + 2 = 5$ , $+ 1 = 6$ (positive). Neither simple input kills both, so choose $x = [- 5 0]$ : Row 1: $- 5 + 0 = - 5$ , $+ 1 = - 4$ ; Row 2: $0 + 0 = 0$ , $- 1 = - 1$ . Now $z = [- 4 - 1]$ . Why hunt for it? We deliberately want a case where both pre-activations are $\leq 0$ to complete the taxonomy.
ReLU: $max (0, - 4) = 0$ , $max (0, - 1) = 0 \Rightarrow a = [00]$ . Why? Every pre-activation is negative, so ReLU clamps them all — the layer emits pure zeros.
Gate the gradient: both $z_{i} \leq 0 \Rightarrow$ mask $= [00]$ , so $δ = g ⊙ mask = [00]$ . Why does this matter? Nothing flows backward — this layer learns nothing on this sample, the worst-case degenerate state.

Verify: with $x = (- 5, 0)$ , $z = (- 4, - 1)$ (both $\leq 0$ ), $a = (0, 0)$ , and $δ = (0, 0)$ . A completely dead layer: zero output, zero gradient. ✓

Recall checkpoint

Recall Why does the zero input NOT give zero output?

Because the layer is affine, not linear — the bias $b$ is added after $Wx$ , so zero input yields $ReLU (b)$ . ::: (Ex 3)

Recall In backprop, why is the ReLU gate applied with

$⊙$ rather than matrix multiply? Each neuron's gate affects only its own gradient — there is no mixing across neurons — so it is a pointwise (element-wise) operation. ::: (Ex 4)

Recall What does ReLU's gradient do exactly at z = 0, and why?

The derivative does not truly exist at the corner; by convention we send the gate to 0 (the strict-inequality branch) because a neuron sitting at 0 output contributed nothing forward, so it fairly contributes nothing backward. ::: (Ex 4, Ex 10)

Recall What single check tells you your transposes in

$δ X^{T}$ and $W^{T} δ$ are right? The gradient must have the same shape as the quantity it differentiates ( $\partial_{W} L$ like $W$ , $\partial_{X} L$ like $X$ ). ::: (Ex 6, Ex 9)

See also 6.4.01-PyTorch-Basics for how these hand-cranked steps become one loss.backward() call, and 6.5.01-Reading-Research-Papers for turning a paper's equations into exactly this kind of worked layer.