Feed-forward network sublayers: Worked examples

Before we start, remember the pieces (no new symbol used before this line):

$x$ = one token's vector, a list of numbers, e.g. $[1, 0, - 1, 2]$ .
$W_{1}$ = the expansion matrix; multiplying $x$ by it gives a longer list.
$b_{1}$ = bias, a fixed list we add on.
$σ$ = the activation — a rule applied number-by-number that bends the line (ReLU, GELU, SiLU).
$W_{2}, b_{2}$ = the projection back to the original length.

The scenario matrix

Every FFN example you could ever face falls in one of these cells. The worked examples below are labelled with the cell they hit.

Cell	Scenario	What can go wrong / what to watch
A	Pre-activation positive	ReLU passes it straight through
B	Pre-activation negative	ReLU kills it → dead neuron
C	Pre-activation exactly zero	Degenerate: $max (0, 0) = 0$ , gradient boundary
D	Full forward pass $d_{m o d e l} \to d_{f f} \to d_{m o d e l}$	Shapes must match at each stage
E	Residual + LayerNorm wrapper	Add then normalize, order matters
F	GELU vs ReLU vs SiLU on same input	Negative values survive smoothly
G	Position-wise check (2 tokens, shared weights)	Same $W$ , different data, no mixing
H	Real-world word problem	Parameter budget / memory
I	Exam twist — "collapse to linear" trap	Why $σ$ is non-optional
J	Limiting behaviour $x \to \pm \infty$	Activations saturate or grow

Example 1 — Cells A, B, C: one neuron, all three sign cases

Forecast: Will each pre-activation be positive, negative, or zero — and which will ReLU zero out?

Step 1 — Compute $z_{A}$ . $z_{A} = (1) (0.5) + (0) (0.2) + (- 1) (- 0.3) + (2) (0.4) + 0.1 = 0.5 + 0 + 0.3 + 0.8 + 0.1 = 1.7$ . Why this step? The dot product $x \cdot w$ is the neuron asking "how much does this pattern appear in $x$ ?" A large positive score means the pattern is strongly present.

Step 2 — Apply ReLU to $z_{A}$ . $ReLU (1.7) = max (0, 1.7) = 1.7$ . (Cell A) Why this step? Positive pre-activation → the neuron "fires" and passes its strength through unchanged.

Step 3 — Compute $z_{B}$ . $z_{B} = (- 2) (0.5) + (0) (0.2) + (1) (- 0.3) + (0) (0.4) + 0.1 = - 1 + 0 - 0.3 + 0 + 0.1 = - 1.2$ . $ReLU (- 1.2) = max (0, - 1.2) = 0$ . (Cell B) Why this step? Negative pre-activation → the pattern is absent (or anti-present), so ReLU outputs $0$ . This neuron is dead for this input, and its gradient is $0$ too — that's the dead-neuron problem.

Step 4 — Compute $z_{C}$ . $z_{C} = (0.2) (0.5) + (- 1) (0.2) + (- 0.3) (- 0.3) + (0) (0.4) + 0.1 = 0.1 - 0.2 + 0.09 + 0 + 0.1 = 0.09$ . Not quite zero, so let's engineer the zero case: with $b = - 0.09$ instead, $z_{C} = 0$ exactly. $ReLU (0) = 0$ . (Cell C) Why this step? Zero is the boundary. ReLU outputs $0$ , but its derivative is undefined here (left slope $0$ , right slope $1$ ); frameworks pick $0$ by convention.

Verify: Sanity — the only sign that survives ReLU is the positive one ( $1.7$ ). The negative ( $- 1.2$ ) and the exact-zero both give $0$ . Dot-product magnitudes are all $\leq ∥ x ∥∥ w ∥$ , e.g. $∥ x_{A} ∥ = 6 \approx 2.449$ , $∥ w ∥ = 0.54 \approx 0.735$ , product $\approx 1.8$ ; our $z_{A} - b = 1.6 < 1.8$ . ✓

Example 2 — Cell D: full forward pass with shape tracking

Forecast: The hidden vector has length 3. Guess how many of its entries survive ReLU.

Step 1 — Expand: $u = x W_{1} + b_{1}$ . $u_{1} = 1 \cdot 1 + (- 1) \cdot 0 + 0 = 1$ $u_{2} = 1 \cdot 0 + (- 1) \cdot 1 + 0 = - 1$ $u_{3} = 1 \cdot 2 + (- 1) \cdot (- 1) + 0.5 = 2 + 1 + 0.5 = 3.5$ So $u = [1, - 1, 3.5]$ . Why this step? Multiplying by $W_{1}$ lifts the 2-D token into a richer 3-D feature space where more patterns can be separated.

Step 2 — Activate: $h = ReLU (u)$ . $h = [max (0, 1), max (0, - 1), max (0, 3.5)] = [1, 0, 3.5]$ . Why this step? Non-linearity — one of the three neurons ( $u_{2}$ ) is switched off, giving the sparsity the parent note mentioned.

Step 3 — Project: $y = h W_{2} + b_{2}$ . $y_{1} = 1 \cdot 1 + 0 \cdot 0 + 3.5 \cdot 1 + 0 = 4.5$ $y_{2} = 1 \cdot (- 1) + 0 \cdot 1 + 3.5 \cdot 0 + 1 = 0$ So $FFN (x) = [4.5, 0]$ . Why this step? We compress back to $d_{m o d e l} = 2$ so the result can be added to the residual stream (next example).

Verify: Shapes: $[1 \times 2] \cdot [2 \times 3] = [1 \times 3]$ ✓, then $[1 \times 3] \cdot [3 \times 2] = [1 \times 2]$ ✓. One of three hidden units died — matches the forecast of ~sparse activation. ✓

Example 3 — Cell E: residual connection + Layer Normalization

Forecast: After adding and normalizing, the two output numbers must have mean $0$ . Guess their signs.

Step 1 — Residual add: $z = x + y$ . $z = [1 + 4.5, - 1 + 0] = [5.5, - 1]$ . Why this step? Residual Connections give the FFN an easy identity path — the network only has to learn the change $y$ , not rebuild $x$ from scratch.

Step 2 — Mean over the 2 features. $μ = \frac{5.5 + ( - 1 )}{2} = 2.25$ . Why this step? Layer Normalization centers each token's own features (unlike batch norm which uses the batch).

Step 3 — Variance and std. Deviations: $5.5 - 2.25 = 3.25$ , $- 1 - 2.25 = - 3.25$ . $Var = \frac{3.2 5 ^{2} + ( - 3.25 ) ^{2}}{2} = 3.2 5^{2} = 10.5625$ , $std = 3.25$ . Why this step? We divide by std so the vector has unit spread — keeps activations from blowing up across 24 stacked layers.

Step 4 — Normalize. $out_{1} = \frac{3.25}{3.25} = 1$ , $out_{2} = \frac{- 3.25}{3.25} = - 1$ . Output $= [1, - 1]$ . Why this step? $(x - μ) / std$ maps the vector to mean $0$ , std $1$ .

Verify: Mean of $[1, - 1] = 0$ ✓ (as forecast). Std $= \frac{1 ^{2} + ( - 1 ) ^{2}}{2} = 1$ ✓. Signs match $z$ 's signs (positive, negative) ✓.

Example 4 — Cell F: ReLU vs GELU vs SiLU on the SAME input

Forecast: For the negative input $- 0.5$ , which of the three give exactly zero, and which give a small negative number?

Step 1 — ReLU. $ReLU (- 0.5) = 0$ ; $ReLU (2) = 2$ . Why this step? Baseline: hard cutoff, no negative information kept.

Step 2 — SiLU at $- 0.5$ . $σ (- 0.5) = \frac{1}{1 + e ^{0.5}} = \frac{1}{1 + 1.64872} = 0.377541$ . $SiLU (- 0.5) = - 0.5 \times 0.377541 = - 0.188770$ . Why this step? SiLU is self-gated — the input scales its own probability of passing, so small negatives leak through with a small negative value (non-zero gradient!).

Step 3 — GELU at $- 0.5$ . $Φ (- 0.5) = 0.308538$ . $GELU (- 0.5) = - 0.5 \times 0.308538 = - 0.154269$ . Why this step? GELU multiplies input by "chance it beats a Gaussian." Negative but non-zero — no dead neuron.

Step 4 — At $x = 2$ . $σ (2) = 0.880797 \Rightarrow SiLU = 1.761594$ . $Φ (2) = 0.977250 \Rightarrow GELU = 1.954500$ . ReLU $= 2$ . Why this step? For large positive input all three approach $x$ — they agree where it matters least.

Verify: At $- 0.5$ : ReLU $= 0$ (only ReLU is exactly zero, as forecast), GELU $= - 0.1543$ , SiLU $= - 0.1888$ , both small and negative ✓. At $x = 2$ all lie just below or at $2$ ✓.

Example 5 — Cell G: position-wise means shared weights, no mixing

Forecast: Does $FFN (x_{2})$ depend in any way on $x_{1}$ ? Guess yes/no.

Step 1 — $FFN (x_{1})$ , $x_{1} = [1, 0]$ . $u = [1 \cdot 1, 1 \cdot 0, 1 \cdot 2 + 0.5] = [1, 0, 2.5]$ ; $h = ReLU = [1, 0, 2.5]$ . $y_{1} = [1 + 0 + 2.5, - 1 + 0 + 0 + 1] = [3.5, 0]$ . Why this step? Same $W_{1}, W_{2}$ as any other token — the network is broadcast across positions.

Step 2 — $FFN (x_{2})$ , $x_{2} = [0, 2]$ . $u = [0, 2, 2 \cdot (- 1) + 0.5] = [0, 2, - 1.5]$ ; $h = ReLU = [0, 2, 0]$ . $y_{2} = [0 \cdot 1 + 2 \cdot 0 + 0 \cdot 1, 0 \cdot (- 1) + 2 \cdot 1 + 0 \cdot 0 + 1] = [0, 3]$ . Why this step? Computed with zero knowledge of $x_{1}$ — the two computations never touch.

Step 3 — Swap invariance. Feeding $[x_{2}, x_{1}]$ produces $[y_{2}, y_{1}] = [[0, 3], [3.5, 0]]$ — just the reverse. Why this step? Confirms the position-wise property $FFN ([x_{1}, x_{2}]) = [FFN (x_{1}), FFN (x_{2})]$ from the parent note.

Verify: $y_{2} = [0, 3]$ used only $x_{2}$ ; no term contains $x_{1}$ ✓ (answer to forecast: no dependence). Output list is a per-token map. ✓

Example 6 — Cell H: real-world parameter budget

Forecast: Guess whether one FFN sublayer is above or below 5 million parameters.

Step 1 — Count $W_{1}$ and $b_{1}$ . $W_{1} : 768 \times 3072 = 2, 359, 296$ ; $b_{1} : 3072$ . Why this step? Every entry of a weight matrix is a learned parameter.

Step 2 — Count $W_{2}$ and $b_{2}$ . $W_{2} : 3072 \times 768 = 2, 359, 296$ ; $b_{2} : 768$ . Why this step? Symmetric projection back — same size matrix.

Step 3 — Sum for one sublayer. $2, 359, 296 \times 2 + 3072 + 768 = 4, 718, 592 + 3, 840 = 4, 722, 432$ . Why this step? Compare to the $\approx 8 d_{m o d e l}^{2}$ estimate: $8 \times 76 8^{2} = 4, 718, 592$ — the estimate drops the tiny biases.

Step 4 — All 12 layers, and RAM. Total $= 12 \times 4, 722, 432 = 56, 669, 184$ params. RAM $= 56, 669, 184 \times 4 = 226, 676, 736$ bytes $\approx 216.2$ MiB. Why this step? FFN alone is a big chunk of BERT-base's ~110 M parameters.

Verify: One sublayer $4, 722, 432 < 5, 000, 000$ ✓ (forecast). $8 d_{m o d e l}^{2} = 4, 718, 592$ matches within the two bias vectors ✓. $226, 676, 736/ 2^{20} = 216.18$ MiB ✓.

Example 7 — Cell I: the "collapse to linear" exam trap

Forecast: Two matrix multiplies with no bend in between — can they really collapse into one? Guess yes/no.

Step 1 — Multiply the weight matrices. $W = W_{1} W_{2} = [1021] [1101] = [1 + 2 0 + 1 0 + 2 0 + 1] = [3121]$ . Why this step? $x (W_{1} W_{2}) = (x W_{1}) W_{2}$ by associativity — no activation to break the chain.

Step 2 — Fold the biases. $b = b_{1} W_{2} + b_{2} = [1, 0] [1101] + [0, 1] = [1, 0] + [0, 1] = [1, 1]$ . Why this step? Distribute: $(x W_{1} + b_{1}) W_{2} + b_{2} = x W_{1} W_{2} + b_{1} W_{2} + b_{2}$ .

Step 3 — Test $x = [3, - 1]$ . Direct-bad: $u = x W_{1} + b_{1} = [3 \cdot 1 + (- 1) \cdot 0 + 1, 3 \cdot 2 + (- 1) \cdot 1 + 0] = [4, 5]$ ; then $u W_{2} + b_{2} = [4 + 5, 5 + 1] = [9, 6]$ . Collapsed: $x W + b = [3 \cdot 3 + (- 1) \cdot 1, 3 \cdot 2 + (- 1) \cdot 1] + [1, 1] = [8, 5] + [1, 1] = [9, 6]$ . Same! Why this step? Proves the collapse: without $σ$ , stacking layers buys zero extra expressive power. That is exactly why the non-linearity is mandatory (ties to Universal Approximation Theorem).

Verify: Both paths give $[9, 6]$ ✓ (forecast: yes, they collapse). This is the whole reason ReLU/GELU/SiLU exist.

Example 8 — Cell J: limiting behaviour of the activations

Forecast: As $z \to - \infty$ , which activation's output goes to $0$ fastest? Guess before computing.

Step 1 — Large positive $z = 10$ . ReLU $= 10$ . SiLU $= 10 \cdot σ (10) = 10 \times 0.9999546 = 9.999546$ . GELU $= 10 \cdot Φ (10) \approx 10 \times 1 = 10$ . Why this step? All three $\to z$ for big positive input — they agree in the "definitely fire" regime.

Step 2 — Large negative $z = - 10$ . ReLU $= 0$ (flat). SiLU $= - 10 \cdot σ (- 10) = - 10 \times 0.0000454 = - 0.000454$ . GELU $= - 10 \cdot Φ (- 10) \approx - 10 \times 7.6 \times 1 0^{- 24} \approx - 7.6 \times 1 0^{- 23}$ . Why this step? GELU decays like a Gaussian tail — astronomically faster than SiLU's exponential tail.

Step 3 — Verdict. As $z \to - \infty$ : ReLU is exactly $0$ ; SiLU $\to 0^{-}$ like $z e^{z}$ ; GELU $\to 0^{-}$ but far faster (Gaussian $\times z$ ). Why this step? Explains why GELU behaves "almost like ReLU" for very negative inputs yet stays smooth near zero.

Verify: $SiLU (- 10) = - 0.000454$ (magnitude tiny) ✓; $GELU (- 10)$ magnitude $\sim 1 0^{- 22} ≪$ SiLU's $\sim 1 0^{- 4}$ ✓ — so GELU decays fastest, as this was the forecast to check. ✓

Recall Self-test

ReLU output for pre-activation $- 1.2$ ? ::: $0$ (dead neuron) Does $FFN (x_{2})$ ever use $x_{1}$ 's data? ::: No — position-wise, independent, shared weights. Why can't we drop the activation $σ$ ? ::: Two linear layers collapse to one ( $x W_{1} W_{2} + \dots$ ), gaining no expressive power. One BERT-base FFN sublayer parameter count? ::: $4, 722, 432 \approx 8 d_{m o d e l}^{2}$ . After residual + LayerNorm, what is the mean of a token's features? ::: $0$ (by construction).

Parent: Feed-forward network sublayers (index 4.1.8) · see also Transformer Encoder Block, Positional Encoding.