Feedforward network — forward pass: Exercises

This page is a self-test ladder for the forward pass. Work each problem on paper first, then open the collapsible solution. Every symbol used here was built in the parent note — but we re-anchor the essentials so you never get stuck on notation.

Recall The 3 objects you need before starting (definitions to memorise)

Every problem below uses the same three things. Picture a stack of layers, information flowing bottom-to-top.

$a^{(ℓ - 1)}$ ::: the activations (outputs) of the layer below — a column of numbers, one per neuron.
$z^{(ℓ)} = W^{(ℓ)} a^{(ℓ - 1)} + b^{(ℓ)}$ ::: the pre-activation: a weighted sum plus a bias. One number per neuron in layer $ℓ$ .
$a^{(ℓ)} = σ^{(ℓ)} (z^{(ℓ)})$ ::: the activation: run each $z$ through the nonlinearity $σ$ element-by-element.

Figure 1 — the four scalar activation shapes. Below, the orange ReLU is flat-then-rising with a sharp corner (the "kink") at $z = 0$ ; the blue sigmoid and green tanh are smooth S-curves pinned between horizontal ceilings/floors (the dotted lines at $\pm 1$ ); the gray dashed line is the do-nothing identity. Keep this picture in mind: "squashing" literally means a curve that flattens toward those dotted rails as $z$ runs off to $\pm \infty$ .

Figure — Feedforward network — forward pass

Level 1 — Recognition

Worked example Solution 1.1

$W^{(ℓ)}$ maps the previous layer's activation (length $n_{ℓ - 1}$ ) into this layer (length $n_{ℓ}$ ). So it has $n_{ℓ}$ rows and $n_{ℓ - 1}$ columns.

$W^{(1)}$ : layer 1 has $n_{1} = 6$ neurons, layer 0 has $n_{0} = 4$ inputs $\Rightarrow$ shape $6 \times 4$ .
$b^{(1)}$ : one bias per neuron in layer 1 $\Rightarrow$ length $6$ .

Check the multiplication: $(6 \times 4) \cdot (4 \times 1) = (6 \times 1)$ , and adding $b^{(1)}$ of length $6$ works. ✓

Worked example Solution 1.2

ReLU passes positives unchanged and clamps everything $\leq 0$ to $0$ — trace the orange curve in Figure 1: flat left of the kink, a straight $45°$ ramp to its right.

$ReLU (- 3.2) = max (0, - 3.2) = 0$
$ReLU (0) = max (0, 0) = 0$
$ReLU (4.5) = max (0, 4.5) = 4.5$

Level 2 — Application

Worked example Solution 2.1

Step (a) pre-activation — a single dot product (row · column) plus bias: $z = 0.5 (2) + 1.0 (- 1) + (- 0.5) (3) + 0.4 = 1.0 - 1.0 - 1.5 + 0.4 = - 1.1$ Step (b) activation: $a = ReLU (- 1.1) = max (0, - 1.1) = 0$ Why $0$ ? The weighted evidence came out net negative, and to the left of the kink (Figure 1) ReLU's output is flat at zero. The neuron "did not fire" — it detected a pattern that is absent for this input.

Worked example Solution 2.2

Sigmoid is applied element-by-element: $σ (z) = \frac{1}{1 + e ^{- z}}$ .

$σ (0) = \frac{1}{1 + e ^{0}} = \frac{1}{2} = 0.500$
$σ (2) = \frac{1}{1 + e ^{- 2}} = \frac{1}{1 + 0.13534} = 0.881$

$a^{(1)} = [0.500 0.881]$ Why these values? At $z = 0$ the term $e^{- z} = 1$ , so top and bottom balance and we sit exactly on the S-curve's midpoint, $0.5$ — this is why sigmoid is centred at $z = 0$ . As $z$ grows, $e^{- z}$ shrinks toward $0$ , so the fraction climbs toward (but never reaches) $1$ ; that is why $σ (2) = 0.881$ is high yet still under the ceiling. Both outputs land inside $(0, 1)$ — the "squashing" you see the blue curve doing in Figure 1.

Worked example Solution 2.3

Layer 1 pre-activation (each row of $W^{(1)}$ dotted with $x$ ): $z^{(1)} = [1 (1) + 0 (2) - 1 (1) + 1 (2)] = [11]$ Layer 1 activation (ReLU): both entries are positive, so we are on the ramp side of the kink and ReLU leaves them unchanged: $a^{(1)} = [11]$ . Layer 2 pre-activation: $z^{(2)} = 2 (1) + 3 (1) + 1 = 6$ Layer 2 activation (linear): $y = z^{(2)} = 6$ . Why linear at the end? A regression output must be free to take any real value; the identity (gray dashed line in Figure 1) applies no squashing, so the raw score $6$ passes straight through as the prediction.

Level 3 — Analysis

Worked example Solution 3.1

Substitute $a^{(1)}$ into $y$ : $y = W^{(2)} (W^{(1)} x + b^{(1)}) + b^{(2)} = W_{eff} W^{(2)} W^{(1)} x + b_{eff} W^{(2)} b^{(1)} + b^{(2)}$ So $W_{eff} = W^{(2)} W^{(1)}$ and $b_{eff} = W^{(2)} b^{(1)} + b^{(2)}$ . Meaning: stacking linear maps gives one linear map — the extra layer bought us nothing. This is exactly why nonlinearity $σ$ is essential.

Worked example Solution 3.2

Layer by layer: $a^{(1)} = [1 (1) + 2 (1) + 1 0 (1) + 1 (1) + 0] = [41]$ , then $y = 1 (4) + 1 (1) + 0 = 5$ . Collapsed: $W_{eff} = W^{(2)} W^{(1)} = [11] [1021] = [13]$ ; $b_{eff} = [11] [10] + 0 = 1$ . Then $y = 1 (1) + 3 (1) + 1 = 5$ . ✓ Same answer.

Worked example Solution 3.3

The neuron outputs $ReLU (2 x - 4)$ . It is dead whenever $2 x - 4 \leq 0$ , i.e. $x \leq 2$ .

Dead region: $x \leq 2$ (output exactly $0$ ).
Active region: $x > 2$ , where output $= 2 x - 4$ rises with slope $2$ .
Kink (the corner) sits at $x = 2$ , where $z = 0$ .

This is the same corner you see on the orange curve in Figure 1, just shifted: the linear part inside ReLU moves the kink from $z = 0$ to $x = 2$ . That hinge is how networks build piecewise-linear approximations of curved aerospace dynamics.

Level 4 — Synthesis

Worked example Solution 4.1

Step 1 — exponentiate each logit: $e^{2} = 7.389, e^{0} = 1.000, e^{- 1} = 0.368$ Step 2 — sum: $S = 7.389 + 1.000 + 0.368 = 8.757$ . Step 3 — divide: $p = 7.389/8.757 1.000/8.757 0.368/8.757 = 0.844 0.114 0.042$ Why exponentiate first? Raw logits can be negative, but probabilities cannot — $e^{z}$ maps every score to a positive number while preserving order (bigger logit → bigger $e^{z}$ ). Dividing by the sum then forces the total to $1.000$ ✓. The largest is class 0, which the index→name table reads as Nominal (probability $0.844$ ).

Worked example Solution 4.2

Layer 1 pre-activation (three dot products): $z^{(1)} = 1 (1) + 1 (- 2) + 0 0 (1) + (- 1) (- 2) + 1 2 (1) + 0 (- 2) - 1 = - 1 31$ Layer 1 activation (tanh, element-wise): $a^{(1)} = tanh (- 1) tanh (3) tanh (1) = - 0.762 0.995 0.762$ Why is $tanh (3)$ almost $1$ ? tanh saturates: once $z$ is a few units from zero the green curve in Figure 1 has already flattened against its $+ 1$ ceiling, so $tanh (3) = 0.995$ is nearly maxed out, while $tanh (- 1)$ and $tanh (1)$ are still on the sloped middle and stay symmetric ( $\mp 0.762$ ). Layer 2 pre-activation: $z_{1}^{(2)} = 1 (- 0.762) + 0 (0.995) + 1 (0.762) = 0.000$ $z_{2}^{(2)} = 0 (- 0.762) + 1 (0.995) - 1 (0.762) = 0.233$ Softmax: $e^{0} = 1.000, e^{0.233} = 1.263$ ; sum $= 2.263$ . $p = [1.000/2.263 1.263/2.263] = [0.442 0.558]$ Why so close to a coin-flip? The two logits differ by only $0.233$ ; since $e^{z}$ is a gentle multiplier for small gaps, a tiny logit lead becomes just a modest probability lead. Predicted class: 1 = Degraded (probability $0.558$ ).

Level 5 — Mastery

Worked example Solution 5.1

The whole point: one matrix multiply handles the whole batch. Compute $Z = WX + b$ (bias broadcast to each column). $WX = [11 - 1 1] [1002] = [11 - 2 2]$ Add $b = [1 - 1]$ to each column: $Z = [1 + 1 1 - 1 - 2 + 1 2 - 1] = [20 - 1 1]$ Apply ReLU element-wise: $A = [2001]$ Why one multiply for both samples? Each column is processed independently by the same $W$ , so stacking inputs side-by-side lets the hardware compute both passes in a single matrix product — this is exactly how real inference runs a whole batch of sensor frames at once. Column 1 is sample 1's activation $[20]$ ; column 2 is sample 2's $[01]$ .

Worked example Solution 5.2

Claim: softmax is invariant to adding a constant $c$ to every logit, because $\frac{e ^{z_{i} + c}}{\sum _{j} e ^{z_{j} + c}} = \frac{e ^{c} e ^{z_{i}}}{e ^{c} \sum _{j} e ^{z_{j}}} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}} .$ The $e^{c}$ cancels top and bottom. So we may pick $c = - max_{j} z_{j} = - 1002$ to keep exponents $\leq 0$ (no overflow). Shifted logits: $- 2 - 1 0$ . Then $e^{- 2} = 0.135, e^{- 1} = 0.368, e^{0} = 1.000$ ; sum $= 1.503$ . $p = 0.135/1.503 0.368/1.503 1.000/1.503 = 0.090 0.245 0.665$ Why this is safe and exact: the cancellation above is algebraic, not an approximation — the probabilities are literally unchanged, but every exponent is now $\leq 0$ so $e^{z} \in (0, 1]$ and can never overflow.

Worked example Solution 5.3

Step 1 — hidden pre-activations (each row of $W^{(1)}$ dotted with $x$ , plus bias): $z_{1} = - 1 (1) - 1 (2) - 1 = - 4, z_{2} = - 2 (1) + 0 (2) - 3 = - 5$ Step 2 — activation (ReLU): both are $\leq 0$ , so we sit on the flat left side of the orange curve (Figure 1) and each clamps to zero: $a^{(1)} = [max (0, - 4) max (0, - 5)] = [00]$ Step 3 — output. With every hidden activation zero, the output pre-activation is $z^{(2)} = W^{(2)} [00] + b^{(2)} = W^{(2)} 0 + b^{(2)} = b^{(2)}$ The weighted-sum term $W^{(2)} 0$ is the zero vector no matter what numbers sit in $W^{(2)}$ , so the prediction equals the output bias alone. Why this matters (the "dying ReLU" failure): overly negative biases push every neuron below its kink; once a neuron outputs $0$ for all realistic inputs it is stuck (its weights stop influencing the output), and if a whole layer dies the network freezes at a constant $b^{(2)}$ and ignores its sensors entirely. Fix in practice: careful bias initialisation, input normalisation, or leaky-ReLU which keeps a small slope on the negative side.