Feed-forward network sublayers: Exercises — AI-ML deep dive

Before we start, a two-line refresher on the notation the parent note uses, so line one makes sense:

Level 1 — Recognition

L1.1 — Dimension of each matrix

The FFN uses two weight matrices $W_{1}$ and $W_{2}$ . Given $d_{model} = 512$ and $d_{f f} = 2048$ , state the shape (rows $\times$ columns) of each.

Recall Solution

A weight matrix that turns a vector of length $a$ into a vector of length $b$ must be shaped $a \times b$ (you multiply the row vector $x$ of length $a$ on the left, $x W$ , and the number of columns becomes the new length).

$W_{1}$ : input $512 \to$ hidden $2048$ , so $W_{1} \in R^{512 \times 2048}$ .
$W_{2}$ : hidden $2048 \to$ output $512$ , so $W_{2} \in R^{2048 \times 512}$ .

L1.2 — Which step adds non-linearity?

In $FFN (x) = σ (x W_{1} + b_{1}) W_{2} + b_{2}$ , exactly one operation makes the whole thing non-linear. Name it, and say what would happen to the network if you removed it.

Recall Solution

The activation $σ$ (ReLU / GELU / SiLU) is the only non-linear part; matrix multiplies and bias additions are all linear (a straight-line map: doubling the input doubles the output). Remove $σ$ and you get $x W_{1} W_{2} + (b_{1} W_{2} + b_{2})$ . Two matrices multiplied together are just one matrix $W_{1} W_{2}$ , so the FFN collapses into a single linear layer — no curved decision boundaries at all.

L1.3 — Position-wise recall

Does the FFN at position 5 read data from position 6? ::: No — the FFN is position-wise; it applies the same weights to each token independently, mixing nothing across positions.

Level 2 — Application

L2.1 — Compute a single hidden unit with ReLU

Let $x = [1, 0, - 1, 2]$ and let the first column of $W_{1}$ be $[0.5, 0.2, - 0.3, 0.1]^{⊤}$ with bias $b_{1} [1] = 0.1$ . Compute $h_{1} = ReLU (x \cdot W_{1} [:, 1] + b_{1} [1])$ .

Recall Solution

The dot product $x \cdot W_{1} [:, 1]$ multiplies matching entries and sums — it measures "how much does $x$ point along this weight pattern?" $1 (0.5) + 0 (0.2) + (- 1) (- 0.3) + 2 (0.1) = 0.5 + 0 + 0.3 + 0.2 = 1.0$ Add bias: $1.0 + 0.1 = 1.1$ . Then $ReLU (1.1) = max (0, 1.1) = 1.1$ .

L2.2 — A negative pre-activation

Same $x = [1, 0, - 1, 2]$ , but now the weight column is $[- 0.4, 0.5, 0.2, - 0.6]^{⊤}$ with bias $- 0.1$ . Compute the ReLU output.

Recall Solution

Dot product: $1 (- 0.4) + 0 (0.5) + (- 1) (0.2) + 2 (- 0.6) = - 0.4 + 0 - 0.2 - 1.2 = - 1.8$ . Add bias: $- 1.8 - 0.1 = - 1.9$ . Then $ReLU (- 1.9) = max (0, - 1.9) = 0$ . This neuron is dead for this input — the pattern it looks for is absent.

L2.3 — Compare ReLU vs SiLU at one point

Compute $ReLU (- 1.9)$ and $SiLU (- 1.9)$ where $SiLU (x) = \frac{x}{1 + e ^{- x}}$ . Which one still passes a (small) non-zero signal?

Recall Solution

$ReLU (- 1.9) = 0$ exactly — a hard cutoff, gradient zero, no learning signal. For SiLU: $σ (- 1.9) = \frac{1}{1 + e ^{1.9}} = \frac{1}{1 + 6.6859} = 0.13010$ , so $SiLU (- 1.9) = (- 1.9) (0.13010) = - 0.24720 .$ SiLU is the one that still passes a small non-zero value, so gradients can flow — look at the orange curve in the figure dipping gently below zero instead of snapping flat.

Level 3 — Analysis

L3.1 — Parameter count of one FFN

For $d_{model} = 768$ , $d_{f f} = 3072$ , count every learnable parameter in the FFN (both weight matrices and both bias vectors).

Recall Solution

Weights: $W_{1}$ has $768 \times 3072$ and $W_{2}$ has $3072 \times 768$ entries. $768 \times 3072 = 2, 359, 296, 3072 \times 768 = 2, 359, 296.$ Biases: $b_{1}$ has $3072$ , $b_{2}$ has $768$ . Total: $2, 359, 296 \times 2 + 3072 + 768 = 4, 718, 592 + 3840 = 4, 722, 432$ parameters. Note the weights dominate; the $\approx 8 d_{model}^{2} = 8 (768)^{2} = 4, 718, 592$ rule matches the weight part exactly.

L3.2 — Why 4× and not equal width?

Explain, using the idea of ReLU sparsity, why the hidden layer is made 4× wider rather than the same width as $d_{model}$ .

Recall Solution

If $d_{f f} = d_{model}$ , the FFN is an information bottleneck: no more feature slots than inputs, so complex combinations ("positive sentiment and past tense") can't get their own dedicated neuron. ReLU zeroes out roughly half its neurons for any given input (sparsity $\approx 50%$ ). So of $d_{f f} = 4 d_{model}$ neurons, only about $2 d_{model}$ are effectively active per token. Expanding 4× keeps the live capacity at about $2 \times$ the input width — enough room to represent rich feature combinations, while the second matrix compresses them back to fit the residual stream.

L3.3 — Compute a full tiny FFN forward pass

Let $d_{model} = 2$ , $d_{f f} = 2$ , activation ReLU. Given $x = [1, - 2], W_{1} = [1 - 1 02], b_{1} = [0, 1], W_{2} = [20 - 1 1], b_{2} = [0, 0] .$ Compute $FFN (x)$ .

Recall Solution

Step 1 — expand. $x W_{1}$ : with row vector $x = [1, - 2]$ , $x W_{1} = [1 \cdot 1 + (- 2) \cdot (- 1), 1 \cdot 0 + (- 2) \cdot 2] = [1 + 2, 0 - 4] = [3, - 4] .$ Add $b_{1} = [0, 1]$ : $[3, - 3]$ . Step 2 — activate. $ReLU ([3, - 3]) = [3, 0]$ . Step 3 — project. $h W_{2}$ with $h = [3, 0]$ : $[3 \cdot 2 + 0 \cdot 0, 3 \cdot (- 1) + 0 \cdot 1] = [6, - 3] .$ Add $b_{2} = [0, 0]$ : $[6, - 3]$ .

Level 4 — Synthesis

L4.1 — FFN inside the residual + LayerNorm block

Given a token's attention output $x^{'} = [2, 0]$ and the FFN result from L3.3's style equals $y = [- 1, 1]$ , the block computes $LayerNorm (x^{'} + y)$ . First form $x^{'} + y$ , then apply LayerNorm with $γ = [1, 1]$ , $β = [0, 0]$ , $ϵ = 0$ . (LayerNorm subtracts the mean and divides by the standard deviation across the feature dimension.)

Recall Solution

Add (residual): $x^{'} + y = [2 - 1, 0 + 1] = [1, 1]$ . Mean: $μ = \frac{1 + 1}{2} = 1$ . Variance: $σ^{2} = \frac{( 1 - 1 ) ^{2} + ( 1 - 1 ) ^{2}}{2} = 0$ , so std $= 0$ . Normalize: each entry is $\frac{1 - 1}{0 + 0}$ — a $0/0$ ! With $ϵ = 0$ this is undefined. Lesson: A degenerate input (all features equal) is exactly why LayerNorm always keeps a tiny $ϵ > 0$ . With, say, $ϵ = 1 0^{- 5}$ : denominator $= 1 0^{- 5} \approx 0.003162$ , numerator $= 0$ , so the normalized output is $[0, 0]$ , then scaled/shifted by $γ, β$ gives $[0, 0]$ . See Layer Normalization and Residual Connections for why this add-then-normalize order stabilizes deep stacks.

L4.2 — Why FFN ≈ 1×1 convolution

Argue in words why applying the FFN to a whole sequence $X = [x_{1}, \dots, x_{n}]$ is equivalent to a 1×1 convolution over the sequence, and why this makes it fully parallel.

Recall Solution

A 1×1 convolution slides a filter that looks at one position at a time (kernel width 1) and applies the same weights everywhere. The FFN does exactly this: the same $W_{1}, b_{1}, W_{2}, b_{2}$ are applied to each $x_{i}$ with no neighbor mixing — $FFN (X) = [FFN (x_{1}), \dots, FFN (x_{n})]$ . Because each position's computation is independent and uses shared weights, all $n$ positions can be computed simultaneously as a single big matrix multiply — perfect parallelism, unlike anything that needs sequential neighbor access. This connects to the Universal Approximation Theorem: each position gets its own two-layer approximator.

Level 5 — Mastery

L5.1 — GELU vs its tanh approximation

The exact GELU is $GELU (x) = x Φ (x)$ where $Φ$ is the standard-normal CDF; the tanh approximation is $GELU (x) \approx 0.5 x (1 + tanh [\frac{2}{π} (x + 0.044715 x^{3})]) .$ Evaluate the tanh approximation at $x = 1$ and report the value.

Recall Solution

Inner term: $2/ π = 0.797885$ . Cubic bit: $x + 0.044715 x^{3} = 1 + 0.044715 = 1.044715$ . Argument of tanh: $0.797885 \times 1.044715 = 0.833562$ . $tanh (0.833562) = 0.682223$ . Then $0.5 (1) (1 + 0.682223) = 0.5 \times 1.682223 = 0.841112$ . The exact GELU at $x = 1$ is $\approx 0.841345$ — the approximation matches to about $3$ decimals, as the near-overlapping curves in the figure show.

L5.2 — Fraction of block parameters in the FFN

For one encoder block with attention costing $\approx 4 d_{model}^{2}$ parameters and FFN costing $\approx 8 d_{model}^{2}$ , compute the fraction of the block's parameters (ignoring LayerNorm) that live in the FFN. Then confirm with $d_{model} = 768$ .

Recall Solution

Fraction $= \frac{8 d _{model}^{2}}{4 d _{model}^{2} + 8 d _{model}^{2}} = \frac{8}{12} = \frac{2}{3} \approx 0.6667$ . The $d_{model}^{2}$ cancels, so the answer is width-independent: two-thirds of each block's parameters are in the FFN. This confirms the parent note's claim that the FFN is where most learning happens and attention is "the router." Numerically for $768$ : FFN $= 8 (768)^{2} = 4, 718, 592$ , attention $= 4 (768)^{2} = 2, 359, 296$ , fraction $= 4, 718, 592/7, 077, 888 = 0.6667$ . ✓

L5.3 — Design a bottleneck-free FFN under a parameter budget

You are given a budget of exactly $2, 000, 000$ weight parameters for a single FFN (ignore biases) with $d_{model} = 500$ . The two weight matrices together cost $2 d_{model} d_{f f}$ . Find the largest integer expansion factor $r = d_{f f} / d_{model}$ you can afford, and state whether it reaches the usual $r = 4$ .

Recall Solution

Weight cost $= 2 d_{model} d_{f f} = 2 (500) (500 r) = 500, 000 r$ . Budget: $500, 000 r \leq 2, 000, 000 \Rightarrow r \leq 4$ . The largest integer is $r = 4$ , which exactly reaches the standard $4 \times$ rule ( $d_{f f} = 2000$ , weight cost $= 2, 000, 000$ ). Any budget below $2, 000, 000$ would force a narrower, more bottlenecked hidden layer.