Worked examples — Forward propagation computation

Before anything: a quick reminder of the two symbols we lean on, in plain words.

Recall What do

$z$ and $a$ mean again? $z$ (pre-activation) ::: the weighted sum of inputs plus bias, before bending — a plain number. $a$ (activation) ::: the number after bending through $\varphi$ — what gets passed forward.

---

The scenario matrix

Every forward-pass exercise is really one of these case classes. Think of it as a checklist: each worked example below fills one or more cells.

#	Case class	What is being stressed	Example
C1	Positive & negative pre-activations with ReLU	the "gate" behaviour, $\max (0,z)$ on both signs	Ex 1
C2	Zero input vector ($\mathbf{x}=0$)	does the network still output something? role of bias	Ex 2
C3	Sigmoid saturation (very large / very negative $z$)	limiting values $\to 1$ and $\to 0$	Ex 3
C4	tanh with mixed signs	odd symmetry, output range $(-1,1)$	Ex 4
C5	Batch of several inputs at once ($N>1$)	matrix form, bias broadcast, shape legality	Ex 5
C6	Softmax — normal + degenerate (all-equal logits)	probabilities summing to 1, uniform case, shift-invariance	Ex 6
C7	Real-world word problem (spam scorer)	translating a story into $W,\mathbf{b},\varphi$	Ex 7
C8	Exam twist: "linear-only net" collapse	proving depth is wasted without $\varphi$	Ex 8

We now clear the board one cell at a time.

---

Ex 1 — ReLU on both signs (C1)

Forecast: guess the sign of each neuron's $z$ before computing. Which neuron do you think ReLU will switch off?

Step 1 — Compute $z_1$ (top neuron). Why this step? Row 1 of $W$ dotted with $\mathbf{x}$ is exactly neuron 1's weighted sum (the parent's "row $j$ = neuron $j$" rule). $z_1=1\cdot 3+2\cdot (-1)+(-4)=3-2-4=-3$

Step 2 — Compute $z_2$ (bottom neuron). Why? Same recipe, row 2. $z_2=(-1)\cdot 3+0\cdot (-1)+5=-3+0+5=2$

Step 3 — Bend through ReLU. Why? ReLU clips negatives to 0. A negative $z$ means the neuron's evidence didn't clear its threshold, so it stays silent. $\mathbf{a}=\left[\begin{array}{c}\max (0,-3)\\ \max (0,2)\end{array}\right]=\left[\begin{array}{c}0\\ 2\end{array}\right]$

Look at the figure: the top neuron's $z=-3$ lands on the flat part of the ReLU (output 0, the gate is shut); the bottom neuron's $z=2$ lands on the sloped part (output passes through unchanged).

Verify: ReLU never returns a negative number — both outputs are $\ge 0$. ✓ Neuron 1 was killed exactly because its $z<0$, matching our forecast if you guessed the top one dies.

---

Ex 2 — The zero input vector (C2)

Forecast: if you feed a network nothing, does it output zero? (Trap: no.)

Step 1 — Weighted sum vanishes. Why? Every product $w_i{x}_i$ has $x_i=0$, so the entire sum $W\mathbf{x}=0$. Only the bias survives. $W\mathbf{x}=\left[\begin{array}{c}0\\ 0\end{array}\right],\mathbf{z}=0+\mathbf{b}=\left[\begin{array}{c}-4\\ 5\end{array}\right]$

Step 2 — Bend. Why? This is the whole point of bias: it lets a neuron fire even with zero input. $\mathbf{a}=\left[\begin{array}{c}\max (0,-4)\\ \max (0,5)\end{array}\right]=\left[\begin{array}{c}0\\ 5\end{array}\right]$

Verify: With $\mathbf{x}=0$ the pre-activation must equal $\mathbf{b}$ exactly — and $\mathbf{z}=(-4,5)=\mathbf{b}$. ✓ This is why biases exist: a bias-free net always outputs $\varphi (0)$ for zero input, unable to shift the decision boundary off the origin.

---

Ex 3 — Sigmoid saturation, both extremes (C3)

The sigmoid squashes any real number into $(0,1)$: $\sigma (z)=\frac{1}{1+e^{-z}}$

Forecast: where does $\sigma$ send a huge positive number? A huge negative one? And exactly $0$?

Step 1 — $z=0$. Why start here? $z=0$ is the pivot; $e^0=1$ so the fraction is $\frac{1}{1+1}$. $\sigma (0)=\frac{1}{1+1}=0.5$

Step 2 — $z=10$ (large positive). Why? To see the upper limiting value. Here $e^{-10}\approx 0.0000454$, tiny. $\sigma (10)=\frac{1}{1+0.0000454}\approx 0.99995$

Step 3 — $z=-10$ (large negative). Why? The lower limit. Now $e^{-(-10)}=e^{10}\approx 22026$, huge. $\sigma (-10)=\frac{1}{1+22026}\approx 0.0000454$

The figure shows the S-curve flattening at both ends — the shaded "saturation zones." There the curve is almost horizontal, so its slope (gradient) is nearly zero. That flatness is the seed of vanishing gradients: in a saturated region the neuron barely responds to changes in $z$.

Verify: $\sigma$ must stay strictly inside $(0,1)$ and satisfy the symmetry $\sigma (-z)=1-\sigma (z)$. Check: $\sigma (10)+\sigma (-10)\approx 0.99995+0.0000454=1.0000$. ✓

---

Ex 4 — tanh with mixed signs (C4)

$\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$ Unlike sigmoid, tanh outputs live in $(-1,1)$ and it is odd: $\tanh (-z)=-\tanh (z)$.

Forecast: because tanh is odd, guess the relationship between the two outputs before computing.

Step 1 — Compute $\tanh (1)$. Why? Directly from the formula with $e^1=2.71828$, $e^{-1}=0.36788$. $\tanh (1)=\frac{2.71828-0.36788}{2.71828+0.36788}=\frac{2.35040}{3.08616}\approx 0.76159$

Step 2 — Compute $\tanh (-1)$. Why? Use oddness — no fresh arithmetic needed. $\tanh (-1)=-\tanh (1)\approx -0.76159$

Step 3 — Assemble. $\mathbf{a}\approx \left[\begin{array}{c}0.76159\\ -0.76159\end{array}\right]$

Verify: Both outputs lie in $(-1,1)$ ✓, and they are exact negatives of each other, confirming oddness ✓ — matching the forecast. This centred-around-zero behaviour is why tanh is often preferred over sigmoid in hidden layers; see Activation functions.

---

Ex 5 — A whole batch at once (C5)

Forecast: first predict the shape of $Z$ before any numbers. (Hint: $W$ is $2\times 2$, $X$ is $2\times 3$.)

Step 1 — Shape sanity. Why first? Illegal shapes are the #1 forward-pass bug. $W$ is $2\times 3$? No — $2\times 2$. $X$ is $2\times 3$. The product $WX$ is $(2\times 2)(2\times 3)=2\times 3$. So $Z$ is $2\times 3$: one column per example.

Step 2 — Matrix multiply $WX$. Why? $W$ is the identity here, so it just copies $X$ — the cleanest way to isolate what the bias does. $WX=X=\left[\begin{array}{ccc}2& 0& 5\\ 3& 1& -4\end{array}\right]$

Step 3 — Broadcast the bias. Why broadcast? $\mathbf{b}$ is $2\times 1$ but $Z$ is $2\times 3$; we add the same bias to every column (every example gets the same neuron offsets). $Z=\left[\begin{array}{ccc}2-1& 0-1& 5-1\\ 3-1& 1-1& -4-1\end{array}\right]=\left[\begin{array}{ccc}1& -1& 4\\ 2& 0& -5\end{array}\right]$

Step 4 — ReLU element-wise. Why? Clip each negative to 0, independently. $A=\left[\begin{array}{ccc}\max (0,1)& \max (0,-1)& \max (0,4)\\ \max (0,2)& \max (0,0)& \max (0,-5)\end{array}\right]=\left[\begin{array}{ccc}1& 0& 4\\ 2& 0& -5\to 0\end{array}\right]=\left[\begin{array}{ccc}1& 0& 4\\ 2& 0& 0\end{array}\right]$

Verify: Column 2 came from input $(0,1)$: $z=(0-1,1-1)=(-1,0)\to (0,0)$ ✓. Every entry of $A$ is $\ge 0$ ✓. Shape is $2\times 3$, exactly as forecast ✓. This is the batch machinery of Matrix multiplication that makes GPUs efficient.

---

Ex 6 — Softmax: normal case AND the degenerate uniform case (C6)

$\text{softmax}(z{)}_k=\frac{e^{z_k}}{{\sum }_j{e}^{z_j}}$

Forecast: for part (b), what must the answer be if all three logits are identical? Guess before computing.

Step 1 — Part (a), exponentiate. Why? Exponentials are positive & monotonic, turning scores into comparable positive weights. $e^1=2.71828,e^0=1,e^{-1}=0.36788,\text{sum}=4.08616$

Step 2 — Normalize. Why divide by the sum? So the three outputs add to exactly 1 — a valid probability distribution. $\hat{y}=\left[\begin{array}{c}2.71828/4.08616\\ 1/4.08616\\ 0.36788/4.08616\end{array}\right]\approx \left[\begin{array}{c}0.6652\\ 0.2447\\ 0.0900\end{array}\right]$

Step 3 — Part (b), the degenerate case. Why does this matter? Equal logits carry zero information about which class wins. Each $e^5$ is identical, so: $\hat{y}=\frac{e^5}{3e^5}\text{ for each}=\frac{1}{3}\approx 0.3333\text{(each)}$

The figure contrasts the two bar charts: peaked on the left, perfectly flat (uniform) on the right.

Verify: (a) $0.6652+0.2447+0.0900=0.9999\approx 1$ ✓. (b) $3\times \frac{1}{3}=1$ ✓, and softmax is shift-invariant — subtracting 5 from every logit gives $(0,0,0)$, whose softmax is also $(\frac{1}{3},\frac{1}{3},\frac{1}{3})$, confirming the answer doesn't depend on the value 5, only on the differences. See Softmax and cross-entropy loss.

---

Ex 7 — Word problem: a tiny spam scorer (C7)

Forecast: more caps and more links push toward spam. With these counts, do you expect $z>0$ or $z<0$?

Step 1 — Translate the story into $z$. Why? Each weight is "how suspicious this feature is per unit." The bias $-3$ is the neuron's baseline scepticism (needs enough evidence before firing). $z=0.6\cdot 4+1.0\cdot 2+(-3)=2.4+2-3=1.4$

Step 2 — Bend through sigmoid. Why sigmoid here? We want a probability in $(0,1)$, and this is the output neuron. $\sigma (1.4)=\frac{1}{1+e^{-1.4}}=\frac{1}{1+0.24660}\approx 0.8022$

Step 3 — Decide. $0.8022>0.5$, so flag as spam.

Verify: Sanity by units — $z$ is dimensionless (a score), $\sigma$ maps it to a probability in $(0,1)$; $0.8022$ is inside that range ✓. Cross-check the threshold: $\sigma (z)>0.5⟺z>0$, and indeed $z=1.4>0$ ✓, matching the forecast.

---

Ex 8 — Exam twist: the linear-only collapse (C8)

Forecast: the parent claimed "stacked linear layers collapse." Predict: will the two-layer output differ from some one-layer output? (Trap: it can't.)

Step 1 — Forward pass the long way. Why? To get a concrete number to compare against. ${\mathbf{z}}^{[1]}=W^{[1]}\mathbf{x}+{\mathbf{b}}^{[1]}=\left[\begin{array}{c}1\cdot 2+2\cdot 3\\ 0\cdot 2+1\cdot 3\end{array}\right]+\left[\begin{array}{c}1\\ 0\end{array}\right]=\left[\begin{array}{c}8+1\\ 3\end{array}\right]=\left[\begin{array}{c}9\\ 3\end{array}\right]$ Identity activation: ${\mathbf{a}}^{[1]}={\mathbf{z}}^{[1]}=\left[\begin{array}{c}9\\ 3\end{array}\right]$. $\hat{y}=W^{[2]}{\mathbf{a}}^{[1]}+b^{[2]}=(1\cdot 9+(-1)\cdot 3)+2=6+2=8$

Step 2 — Collapse algebraically. Why? To prove the collapse in general, substitute layer 1 into layer 2: $\hat{y}=W^{[2]}(W^{[1]}\mathbf{x}+{\mathbf{b}}^{[1]})+b^{[2]}=\underset{W_{\text{eff}}}{\underbrace{W^{[2]}{W}^{[1]}}}\mathbf{x}+\underset{b_{\text{eff}}}{\underbrace{W^{[2]}{\mathbf{b}}^{[1]}+b^{[2]}}}$

Step 3 — Compute the effective single layer. Why? This is the "one layer that does the same job." $W_{\text{eff}}=\left[\begin{array}{cc}1& -1\end{array}\right]\left[\begin{array}{cc}1& 2\\ 0& 1\end{array}\right]=\left[\begin{array}{cc}1& 1\end{array}\right]$ $b_{\text{eff}}=\left[\begin{array}{cc}1& -1\end{array}\right]\left[\begin{array}{c}1\\ 0\end{array}\right]+2=1+2=3$ So the whole two-layer net is just $\hat{y}=x_1+x_2+3$.

Verify: Plug $\mathbf{x}=(2,3)$ into the collapsed form: $2+3+3=8$ — identical to Step 1 ✓. This is exactly the parent's warning: without a nonlinearity, depth buys nothing — the reason Activation functions and the Universal approximation theorem matter, and the thing Backpropagation later trains.

---

---

Flashcards