Implementing models from scratch: Exercises — AI-ML deep dive

Before we start, let me pin down every symbol so nothing appears unearned.

Definition The symbols we use everywhere

$x$ — a column of input numbers (one data example). If it has $n$ numbers we write $x \in R^{n}$ , read " $x$ lives in $n$ -dimensional space".
$W$ — the weight matrix, a grid of numbers. $W \in R^{m \times n}$ means " $m$ rows, $n$ columns".
$b$ — the bias, a column of $m$ numbers added after the multiply.
$z = Wx + b$ — the pre-activation (the raw output before bending it).
$σ$ — an activation function, a rule applied to each number of $z$ to bend the line into a curve.
$L$ — the loss, a single number saying "how wrong are we" (bigger = worse).
$\frac{\partial L}{\partial W}$ — "how much $L$ changes when we nudge $W$ a tiny bit". This is a gradient (a slope). Whenever you see $\partial$ , read it as "tiny change in".
$δ$ (delta) — shorthand for the gradient arriving at a layer's pre-activation: $δ = \frac{\partial L}{\partial z}$ .
$⊙$ — element-wise multiply: multiply matching positions, no summing.
$α$ — the learning rate, a small positive number controlling step size.

Level 1 — Recognition

Exercise 1.1 — Name the shapes

A dense layer maps a vector of $6$ features to $4$ outputs. State the shapes of $W$ , $b$ , and (for a batch of $10$ examples) the input $X$ and output $Z$ .

Recall Solution

A row of $W$ is one linear function turning $6$ inputs into $1$ output number. We have $4$ outputs, so $4$ rows, each of length $6$ : $W \in R^{4 \times 6} .$ One bias per output neuron: $b \in R^{4 \times 1}$ . Batch convention (features down the rows, examples across the columns): $X \in R^{6 \times 10}$ . Then $Z = WX + b$ : multiplying $(4 \times 6)$ by $(6 \times 10)$ gives $(4 \times 10)$ , so $Z \in R^{4 \times 10} .$ The bias $(4 \times 1)$ is broadcast — copied across all $10$ columns.

Exercise 1.2 — Recall the three roles

Match each object to its job: (a) $W$ , (b) $b$ , (c) $σ$ . Jobs: rotate/scale the input space · shift the output · inject non-linearity.

Recall Solution

$W$ ::: rotates and scales the input space (a linear map).
$b$ ::: shifts the output (moves the decision boundary without changing its tilt).
$σ$ ::: injects non-linearity so stacked layers do not collapse into one big linear layer.

Level 2 — Application

Exercise 2.1 — Compute a forward pass by hand

Let $W = [10 - 2 3], b = [1 - 1], x = [21] .$ Compute $z = Wx + b$ , then $a = ReLU (z)$ where $ReLU (z) = max (0, z)$ .

Recall Solution

Row by row — each row of $W$ dotted with $x$ : $z_{1} = (1) (2) + (- 2) (1) + 1 = 2 - 2 + 1 = 1,$ $z_{2} = (0) (2) + (3) (1) + (- 1) = 0 + 3 - 1 = 2.$ So $z = [12]$ . Both are positive, so ReLU passes them through unchanged: $a = [12] .$

Exercise 2.2 — Gradient of weights

Continuing 2.1, suppose the gradient arriving at the pre-activation is $δ = \frac{\partial L}{\partial z} = [0.5 - 1]$ (single example, batch size $1$ ). Using $\frac{\partial L}{\partial W} = δ x^{T}$ , compute $\frac{\partial L}{\partial W}$ and $\frac{\partial L}{\partial b}$ .

Recall Solution

Why $δ x^{T}$ ? From $z_{i} = \sum_{j} W_{ij} x_{j} + b_{i}$ we get $\frac{\partial z _{i}}{\partial W _{ij}} = x_{j}$ . Chain rule: $\frac{\partial L}{\partial W _{ij}} = δ_{i} x_{j}$ — the outer product of $δ$ (column) with $x$ (row): $\frac{\partial L}{\partial W} = [0.5 - 1] [21] = [0.5 \cdot 2 - 1 \cdot 2 0.5 \cdot 1 - 1 \cdot 1] = [1 - 2 0.5 - 1] .$ Shape check: $(2 \times 1) \times (1 \times 2) = (2 \times 2)$ , same as $W$ ✓. For bias, $\frac{\partial z _{i}}{\partial b _{i}} = 1$ , so $\frac{\partial L}{\partial b} = δ = [0.5 - 1]$ (with batch $> 1$ you would sum across the batch).

Exercise 2.3 — One SGD step

Using $\frac{\partial L}{\partial W}$ from 2.2 and learning rate $α = 0.1$ , apply $W \leftarrow W - α \frac{\partial L}{\partial W}$ .

Recall Solution

$W_{new} = [10 - 2 3] - 0.1 [1 - 2 0.5 - 1] = [1 - 0.1 0 + 0.2 - 2 - 0.05 3 + 0.1] = [0.9 0.2 - 2.05 3.1] .$ We subtract the gradient because the gradient points uphill (toward larger loss); the minus turns us downhill.

Level 3 — Analysis

Exercise 3.1 — Why does the linear model collapse?

Show that two stacked linear layers (no activation) $z_{2} = W_{2} (W_{1} x + b_{1}) + b_{2}$ are equivalent to a single linear layer. Give the effective weight and bias.

Recall Solution

Expand using distributivity of matrix multiplication: $z_{2} = W_{2} W_{1} x + W_{2} b_{1} + b_{2} .$ Define $W^{'} = W_{2} W_{1}$ and $b^{'} = W_{2} b_{1} + b_{2}$ . Then $z_{2} = W^{'} x + b^{'}$ — a single affine layer. Figure below: two straight-line maps compose into one straight-line map; no bend appears anywhere.

Figure — Implementing models from scratch

This is exactly why $σ$ exists: only a bend (non-linearity) between the layers stops this collapse and lets the network draw curved boundaries.

Exercise 3.2 — Dead ReLU

A neuron has pre-activations $z = - 2.5 - 0.1 3.7$ across three examples. The incoming gradient is $\frac{\partial L}{\partial a} = 444$ . Compute the gradient flowing back through ReLU, $\frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} ⊙ (z > 0)$ . Which examples pass gradient?

Recall Solution

ReLU's derivative is a gate: $1$ where $z > 0$ , else $0$ . $(z > 0) = 001 \Rightarrow \frac{\partial L}{\partial z} = 444 ⊙ 001 = 004 .$ Only the third example ( $z = 3.7 > 0$ ) passes gradient. The first two are in ReLU's flat region — no gradient, so the weights feeding them learn nothing from these examples. See the gate picture:

Edge case $z = 0$ : by the parent's convention $ReLU^{'} (0) = 0$ (the rule uses $z > 0$ , strict). At exactly $0$ the true slope is undefined (a corner), and libraries just pick $0$ or $1$ by convention.

Level 4 — Synthesis

Exercise 4.1 — One full training step end-to-end

Single example. Layer 1: $W_{1} = [1101]$ , $b_{1} = 0$ , then ReLU. Layer 2: $W_{2} = [1 - 1]$ , $b_{2} = 0$ . Input $x = [12]$ , target $y = 0$ , loss $L = \frac{1}{2} (\overset{y}{^} - y)^{2}$ . Learning rate $α = 0.1$ . Do forward → backward → update $W_{2}$ .

Recall Solution

Forward. $z_{1} = W_{1} x = [1 \cdot 1 + 0 \cdot 2 1 \cdot 1 + 1 \cdot 2] = [13], a_{1} = ReLU [13] = [13] .$ $\overset{y}{^} = W_{2} a_{1} = 1 \cdot 1 + (- 1) \cdot 3 = - 2.$ $L = \frac{1}{2} (- 2 - 0)^{2} = \frac{1}{2} (4) = 2.$

Backward. Loss derivative: $\frac{\partial L}{\partial y ^} = (\overset{y}{^} - y) = - 2 - 0 = - 2$ . Gradient for $W_{2}$ (using $δ a_{1}^{T}$ with $δ = - 2$ ): $\frac{\partial L}{\partial W _{2}} = (- 2) [13] = [- 2 - 6] .$

Update. $W_{2} \leftarrow [1 - 1] - 0.1 [- 2 - 6] = [1 + 0.2 - 1 + 0.6] = [1.2 - 0.4] .$ Sanity: the prediction was too low ( $- 2$ vs target $0$ ), so we push $W_{2}$ toward larger output — both entries moved up. ✓

Exercise 4.2 — Momentum accumulation

Momentum: $v_{t} = β v_{t - 1} + (1 - β) g_{t}$ , update $θ \leftarrow θ - α v_{t}$ . Wait — the parent's snippet omits the $(1 - β)$ scaling; use the classic form $v_{t} = β v_{t - 1} + g_{t}$ . With $β = 0.9$ , $v_{0} = 0$ , and three identical gradients $g_{1} = g_{2} = g_{3} = 1$ , compute $v_{1}, v_{2}, v_{3}$ .

Recall Solution

$v_{1} = 0.9 (0) + 1 = 1,$ $v_{2} = 0.9 (1) + 1 = 1.9,$ $v_{3} = 0.9 (1.9) + 1 = 2.71.$ Notice the velocity grows past the raw gradient of $1$ : momentum accelerates when successive gradients agree in direction. Its steady-state limit is $\frac{1}{1 - β} = \frac{1}{0.1} = 10$ times the constant gradient — momentum can move up to $10 \times$ faster than plain SGD on a consistent slope.

Level 5 — Mastery

Exercise 5.1 — Batch bias gradient, all-signs case

A layer with $2$ output neurons sees a batch of $3$ examples. The incoming pre-activation gradient is $δ = [10 - 2 3 0.5 - 1] \in R^{2 \times 3} .$ Compute $\frac{\partial L}{\partial b} = \sum_{batch} δ$ (sum across columns, keep shape $2 \times 1$ ).

Recall Solution

Bias is shared across every example, so its total gradient is the sum of each example's contribution — sum each row: $\frac{\partial L}{\partial b} = [1 + (- 2) + 0.5 0 + 3 + (- 1)] = [- 0.5 2] .$ Note we sum (not average) here; whether you divide by batch size is a convention baked into your loss — be consistent (the parent's code divides by batch_size, giving $[- 0.1667 0.6667]$ instead).

Exercise 5.2 — Learning-rate stability limit

For a 1-D quadratic loss $L (θ) = \frac{1}{2} c θ^{2}$ with $c > 0$ , the gradient is $g = c θ$ . Gradient descent gives $θ_{t + 1} = θ_{t} - α c θ_{t} = (1 - α c) θ_{t}$ . Find the exact range of $α$ for which $θ_{t} \to 0$ , and the value of $α$ giving the fastest convergence.

Recall Solution

Each step multiplies $θ$ by the factor $r = (1 - α c)$ . The sequence $θ_{t} = r^{t} θ_{0}$ shrinks to $0$ iff $∣ r ∣ < 1$ : $- 1 < 1 - α c < 1 ⟹ 0 < α c < 2 ⟹ 0 < α < \frac{2}{c} .$

$α = \frac{1}{c}$ gives $r = 0$ : converges in one step (fastest).
$\frac{1}{c} < α < \frac{2}{c}$ : $r$ is negative, so $θ$ oscillates in sign while shrinking.
$α = \frac{2}{c}$ : $r = - 1$ , $θ$ bounces forever, never settling.
$α > \frac{2}{c}$ : $∣ r ∣ > 1$ , $θ$ diverges — this is why too-large learning rates blow up.

Exercise 5.3 — Gradient of a whole mini-batch matmul

Prove the batched weight gradient $\frac{\partial L}{\partial W} = δ X^{T}$ automatically sums over the batch, where $X \in R^{n \times B}$ ( $B$ = batch size) and $δ \in R^{m \times B}$ .

Recall Solution

Write the matrix product entry-wise. The $(i, j)$ entry of $δ X^{T}$ is: $(δ X^{T})_{ij} = \sum_{k = 1}^{B} δ_{ik} (X^{T})_{k j} = \sum_{k = 1}^{B} δ_{ik} X_{j k} .$ Read this: for weight $W_{ij}$ , we take example $k$ 's incoming gradient $δ_{ik}$ times that example's input feature $X_{j k}$ , and sum over all $B$ examples. The single matrix multiply $δ X^{T}$ is the batch sum — no explicit loop needed. That is the whole reason vectorised backprop is fast. (Dividing this by $B$ turns the sum into the mean gradient, matching the parent's code.)

Recall Self-test roundup

Fastest 1-D learning rate for $L = \frac{1}{2} c θ^{2}$ ? ::: $α = 1/ c$ (converges in one step). Backward derivative of ReLU at $z = - 2.5$ ? ::: $0$ (dead: input was in the flat region). Why subtract the gradient in the update? ::: the gradient points uphill; subtracting moves us downhill toward lower loss. Effective weight of two stacked linear layers $W_{2}, W_{1}$ ? ::: $W_{2} W_{1}$ (they collapse into one).

Related: 6.3.04-Optimization-Algorithms · 6.4.01-PyTorch-Basics · 6.5.01-Reading-Research-Papers