Worked examples — Multi-layer perceptron architecture

This page is the "hands dirty" companion to the MLP parent note. There we built the machinery: the neuron equation $z={\sum }_i{w}_i{a}_i+b$, the activation $a=\sigma (z)$, and the three output styles (sigmoid, softmax, linear). Here we run that machinery through every kind of input the topic can hand you — every sign, every degenerate case, every limit — so nothing surprises you later.

Before anything, one thing the parent note used but never pictured: what a single neuron actually is. The figure below lays out the whole flow left-to-right.

---

The scenario matrix

Every example below is tagged with the cell it covers. First, here is what each verbal label means, so the label→example mapping is never ambiguous:

• "all-positive" = every input $a_i\ge 0$ and the resulting $z>0$, so ReLU passes it straight through (the plain, no-surprises case).
• "mixed signs" = the weights and inputs combine to a negative $z$, so ReLU clips it to 0 — the normal on/off switching of a healthy ReLU.
• "zero input" = every $a_i=0$, so only the bias is left.
• "dead / dying ReLU" = $z$ is negative for every possible input, so the neuron is stuck at 0 forever (not just once).
• "saturation (limit)" = $|z|$ is huge, pushing sigmoid to its flat tails near 0 or 1.
• "overflow" = softmax logits so large that $e^z$ would break a computer.
• "degenerate softmax" = all logits equal, forcing a uniform output.

Cell	What makes it tricky	Covered by
A — all-positive	plain forward pass, ReLU stays "on"	Ex 1
B — mixed signs	some contributions negative, ReLU clips a negative $z$ to 0	Ex 2
C — zero input	$a_i=0$, only the bias survives	Ex 3
D — dead / dying ReLU	$z$ always $\le 0$, neuron outputs 0 forever	Ex 4
E — sigmoid saturation (limit)	$	z
F — softmax + numerical overflow	large logits, $e^z$ explodes, need the shift trick	Ex 6
G — degenerate softmax (all equal)	every logit identical $\Rightarrow$ uniform output	Ex 7
H — real-world word problem	build architecture + count parameters	Ex 8
I — exam twist (linear collapse)	remove $\sigma$, prove layers merge into one	Ex 9

---

Example 1 — Cell A: the all-positive warm-up

Forecast: everything here is positive. Guess now — will ReLU change the answer or leave it alone?

1. Compute the pre-activation $z$. $z=(0.5)(2)+(2.0)(1)+(1.5)(3)+1=1+2+4.5+1=8.5$ Why this step? $z$ is the neuron's "vote total" — every input scaled by how much the neuron cares, plus the bias nudge. Everything downstream depends on this one number. Notice every weight and every input is positive, so every term adds — this is exactly what "all-positive" means.

2. Apply ReLU. $a=\max (0,8.5)=8.5$ Why this step? ReLU asks "is the vote positive?" Here yes, so it passes $z$ through untouched.

Verify: $z=8.5>0$, so ReLU is the identity — output equals $z$. Sanity: all four contributions ($1,2,4.5,1$) are positive and simply add up, no clipping possible. Answer $a=8.5$. ✓

---

Example 2 — Cell B: mixed signs, ReLU clips

Forecast: the third weight is $-1$ and its input is $+3$ — that's a big negative pull. Will $z$ survive?

1. Compute $z$. $z=(0.5)(-2)+(2.0)(0)+(-1.0)(3)+1=-1+0-3+1=-3$ Why this step? Negative weight $\times$ positive input $=$ negative contribution. The neuron is voting against this pattern. The signs mix, and the negatives win — that is what "mixed signs" means here.

2. Apply ReLU. $a=\max (0,-3)=0$ Why this step? ReLU asks "positive vote?" — no. So it outputs exactly 0. The neuron stays silent for this input.

Verify: $z=-3<0\Rightarrow$ ReLU floors it to 0. This is the normal, healthy behaviour of ReLU: silent on inputs it dislikes, active on ones it likes. (Contrast with Cell D below, where it's stuck silent forever.) Answer $a=0$. ✓

The figure below plots the ReLU curve and marks both of these cases on it — the orange dot is Example 1's positive $z$ passing straight through, and the magenta dot is this example's negative $z$ getting flattened to 0. Notice the sharp "elbow" at $z=0$ where ReLU switches from "off" (flat at 0) to "on" (rising line).

---

Example 3 — Cell C: zero input, only the bias survives

Forecast: every input is zero. Does the neuron output zero too?

1. Compute $z$. $z=(3)(0)+(-7)(0)+(4)(0)+0.6=0.6$ Why this step? Every weighted term dies because $w_i\cdot 0=0$. Only the bias is left. This is exactly what the bias is for — it lets a neuron have a preferred output even when it hears nothing.

2. Apply sigmoid. $a=\frac{1}{1+e^{-0.6}}=\frac{1}{1+0.5488}\approx 0.6457$ Why this step? Sigmoid squashes any real number into $(0,1)$; here it turns the leftover bias into a probability-like value.

Verify: with all inputs zero the output must equal $\sigma (b)$. Since $b=0.6>0$, the output must be above $0.5$ (sigmoid crosses $0.5$ at $z=0$). Indeed $0.6457>0.5$. ✓

---

Example 4 — Cell D: the dying-ReLU trap

Forecast: can any non-negative input ever make this neuron fire?

1. Write $z$ for a general non-negative input. $z=-4a_1-2a_2-5,a_1\ge 0,a_2\ge 0$ Why this step? To prove "always dead" we can't test one input — we must reason over all allowed inputs at once.

2. Bound $z$ from above. Since $a_1,a_2\ge 0$, the terms $-4a_1$ and $-2a_2$ are $\le 0$. So $z\le -5<0\text{for every allowed input.}$ Why this step? The largest $z$ can ever be is when $a_1=a_2=0$, giving $z=-5$. Bigger inputs only push $z$ more negative.

3. Apply ReLU. $a=\max (0,z)=0\text{always.}$ Why this step? We just proved $z<0$ for every allowed input, and ReLU sends every negative number to 0 — so the output is 0 no matter what, confirming the neuron is dead.

Verify: the neuron's gradient through ReLU is 0 whenever $z<0$, so backprop (see backprop) can never update these weights — it is stuck dead permanently. This is the dying ReLU problem, fixed by Leaky ReLU which leaks a small slope for $z<0$. ✓

---

Example 5 — Cell E: sigmoid saturation (a limit)

Forecast: $\sigma$ lives in $(0,1)$. How close to the edges does $z=\pm 12$ push it?

1. Large positive logit. $\sigma (12)=\frac{1}{1+e^{-12}}=\frac{1}{1+6.14\times 10^{-6}}\approx 0.99999386$ Why this step? $e^{-12}$ is tiny, so the denominator is barely above 1 — output is glued near 1.

2. Large negative logit. $\sigma (-12)=\frac{1}{1+e^{12}}=\frac{1}{1+162754.79}\approx 6.14\times 10^{-6}$ Why this step? Now $e^{12}$ dominates the denominator, crushing the output near 0.

3. The gradient. The sigmoid derivative is ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$. ${\sigma }^{\prime }(12)\approx 0.99999386\times 6.14\times 10^{-6}\approx 6.1\times 10^{-6}\approx 0$ Why this step? When $\sigma$ is near 0 or near 1, the product $\sigma (1-\sigma )$ collapses to $\approx 0$ — the vanishing gradient. Learning stalls in these flat tails.

Verify: limiting behaviour: ${\lim }_{z\to +\infty }\sigma (z)=1$ and ${\lim }_{z\to -\infty }\sigma (z)=0$. Our values $0.99999\dots$ and $\approx 6\times 10^{-6}$ sit right at those limits. The maximum of ${\sigma }^{\prime }$ is $0.25$ at $z=0$; both our gradients are microscopic by comparison. ✓

The figure below shows why: the violet sigmoid curve flattens hard in both tails, and the orange dashed curve (its slope ${\sigma }^{\prime }$) is a tiny bump near $z=0$ that vanishes to nothing at $z=\pm 12$ — the two magenta dots sit exactly where the slope has died.

---

Example 6 — Cell F: softmax with overflow, and the shift trick

Forecast: the logits are huge but their gaps are tiny (1 apart). Guess: will the probabilities be extreme or gentle?

Recall softmax: $\text{softmax}(z_i)=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}$.

1. Shift by the max. Subtract $c={\max }_j{z}_j=1001$ from every logit. $\mathbf{z}-c=[-1,0,-2]$ Why this step? Softmax is unchanged if we subtract the same constant from every logit, because $e^{z_i-c}=e^{z_i}{e}^{-c}$ and the $e^{-c}$ cancels top and bottom. But now the exponents are small and safe — no overflow.

2. Exponentiate the shifted logits (rounded to 4 decimals). $e^{-1}\approx 0.3679,e^0=1,e^{-2}\approx 0.1353$ Why this step? Softmax needs $e^{z_i}$ in the numerator; exponentiating turns each shifted logit into a positive "unnormalized weight" that we can then compare and normalize.

3. Normalize. Sum $\approx 0.3679+1+0.1353=1.5032$ (this total is approximate — it is the sum of the rounded values above, not an exact figure). $\text{softmax}\approx \left[\frac{0.3679}{1.5032},\frac{1}{1.5032},\frac{0.1353}{1.5032}\right]\approx [0.2447,0.6652,0.0900]$ Why this step? Dividing each weight by the total forces the outputs to sum to 1, turning raw scores into a valid probability distribution.

Verify: probabilities sum to $0.2447+0.6652+0.0900=0.9999\approx 1$ (the small shortfall from 1 is pure rounding — with full precision they sum to exactly 1). The middle class (largest logit) wins with $\approx 0.665$, but the gaps are only 1 apart so it's not a landslide — matching the forecast that gentle logit gaps give gentle probabilities. The absolute size (1000) was a red herring; only differences matter. ✓

---

Example 7 — Cell G: degenerate softmax (all logits equal)

Forecast: the network is perfectly undecided. What probability does each class get?

1. Shift by the max ($c=5$): $\mathbf{z}-c=[0,0,0,0]$. Why this step? Same overflow-safe trick; also makes the symmetry obvious.

2. Exponentiate: $e^0=1$ for all four, so the vector is $[1,1,1,1]$, sum $=4$. Why this step? Softmax's numerator is $e^{z_i}$; since every shifted logit is 0, every unnormalized weight equals $e^0=1$ — they are all identical, which is what forces a uniform result.

3. Normalize: $\text{softmax}=\left[\frac{1}{4},\frac{1}{4},\frac{1}{4},\frac{1}{4}\right]=[0.25,0.25,0.25,0.25]$ Why this step? Equal inputs must give equal outputs by symmetry — there is no information to prefer any class.

Verify: general rule — if all $K$ logits are equal, softmax outputs $1/K$ each. Here $1/4=0.25$, and $4\times 0.25=1$. ✓ This is exactly the state an untrained network starts near, and it's the maximum-uncertainty output.

---

Example 8 — Cell H: real-world word problem (build + count)

Forecast: it's regression, not classification — guess the output activation before reading.

1. Choose the output layer. One output neuron, no (linear) activation. Why this step? We predict a real number (0–100), not a probability. Sigmoid/softmax would trap the output in $(0,1)$; a linear output can produce any real value. (Feature scaling belongs to feature engineering.)

2. Lay out the shapes. Input 6 → Hidden1 12 → Hidden2 6 → Output 1. Each weight matrix $W^{(l)}$ is (neurons in $l$) $\times$ (neurons in $l-1$); each layer also has one bias per neuron. Why this step? The parameter count comes directly from these shapes — we can't count weights and biases until we know how many neurons feed into how many, so pinning down each layer's size is the prerequisite for the arithmetic in step 3.

3. Count parameters using $\text{(in}\times \text{out)}+\text{out}$ per layer. $\underset{84}{\underbrace{(6\cdot 12+12)}}+\underset{78}{\underbrace{(12\cdot 6+6)}}+\underset{7}{\underbrace{(6\cdot 1+1)}}=84+78+7=169$ Why this step? Total learnable parameters $=$ all weights $+$ all biases. This decides model capacity and overfitting risk — where dropout would help.

Verify: layer breakdown $84+78+7=169$ parameters, with output activation linear (identity). Units check: input features feed a chain of matrix multiplies ending in a single scalar score — dimensionally a real number, correct for a 0–100 score. ✓

---

Example 9 — Cell I: the exam twist (why layers collapse without $\sigma$)

Forecast: three matrices in a row with no non-linearity — how many effective layers remain?

1. Set up (ignore biases for clarity). Let $W_1=\left[\begin{array}{cc}1& 2\\ 0& 1\end{array}\right],W_2=\left[\begin{array}{cc}2& 0\\ 1& 3\end{array}\right],W_3=\left[\begin{array}{cc}1& 1\end{array}\right].$ Input $\mathbf{x}$, and each layer just does ${\mathbf{a}}^{(l)}=W_l{\mathbf{a}}^{(l-1)}$ (no $\sigma$). Why this step? Biases only add a constant shift and don't change the "do the layers merge?" question; dropping them keeps the algebra clean so the collapse is easy to see, and the same argument works with biases included.

2. Compose the matrices. Because matrix multiplication is associative, $\hat{y}=W_3(W_2(W_1\mathbf{x}))=(W_3{W}_2{W}_1)\mathbf{x}=W_{\text{eq}}\mathbf{x}.$ Why this step? Without a non-linearity between them, the layers merge — three matrices are equivalent to their single product $W_{\text{eq}}$.

3. Multiply the inner two matrices first. $W_2{W}_1=\left[\begin{array}{cc}2& 0\\ 1& 3\end{array}\right]\left[\begin{array}{cc}1& 2\\ 0& 1\end{array}\right]=\left[\begin{array}{cc}2\cdot 1+0\cdot 0& 2\cdot 2+0\cdot 1\\ 1\cdot 1+3\cdot 0& 1\cdot 2+3\cdot 1\end{array}\right]=\left[\begin{array}{cc}2& 4\\ 1& 5\end{array}\right].$ Why this step? We collapse two of the three layers into one $2\times 2$ matrix; doing the inner product first keeps the numbers small and matches the left-to-right flow of the network.

4. Multiply by $W_3$ to get the single equivalent layer. $W_{\text{eq}}=W_3(W_2{W}_1)=\left[\begin{array}{cc}1& 1\end{array}\right]\left[\begin{array}{cc}2& 4\\ 1& 5\end{array}\right]=\left[\begin{array}{cc}1\cdot 2+1\cdot 1& 1\cdot 4+1\cdot 5\end{array}\right]=\left[\begin{array}{cc}3& 9\end{array}\right].$ Why this step? This finishes the collapse: all three "layers" reduce to one $1\times 2$ weight row, which we can read off directly as a plain linear formula.

5. Read off the formula. With $W_{\text{eq}}=\left[\begin{array}{cc}3& 9\end{array}\right]$ and $\mathbf{x}=[x_1,x_2{]}^T$, $\hat{y}=3x_1+9x_2.$ Why this step? This is the punchline — the "deep" three-layer network computes exactly the same thing as a single linear layer with weights $3$ and $9$.

Verify: the whole "deep" network computes $\hat{y}=3x_1+9x_2$ — a single linear layer. Spot-check with $\mathbf{x}=[1,1{]}^T$: the layer-by-layer path gives $W_1\mathbf{x}=[3,1{]}^T$, then $W_2[3,1{]}^T=[6,6{]}^T$, then $W_3[6,6{]}^T=12$; and the collapsed formula gives $3(1)+9(1)=12$ — identical. That's why the parent note insists on $\sigma$: it breaks this collapse and is what separates an MLP from the linear single perceptron. ✓

---

Recall

What survives at a neuron when every input is zero? ::: Only the bias $b$; output is $\sigma (b)$. Why does subtracting ${\max }_j{z}_j$ leave softmax unchanged? ::: The common factor $e^{-c}$ cancels in numerator and denominator; only logit differences matter. Why is a ReLU neuron with all-negative reachable $z$ "dead"? ::: Its gradient is 0 for $z<0$, so backprop can never update it and its output stays stuck at 0 forever. Without any activation function, what does a 3-layer network equal? ::: One linear layer, $W_3{W}_2{W}_1$.