Visual walkthrough — Feed-forward network sublayers

We assume you know only one thing: multiplying two numbers, and adding. Everything else — what a "weight", a "matrix", a "ReLU", a "hidden layer" mean — is built below, in pictures.

Step 1 — One input, one weight: what "$w\cdot x$" is even doing

WHAT. Start with the smallest possible piece. We have one number coming in — call it $x$ (the "input"). We have one dial we can turn — call it $w$ (a "weight"). The machine's whole job at this stage is to compute $w\cdot x$: stretch or shrink the input.

WHY. Before any transformer, before any matrix, we must understand what a single weight means. A weight is a volume knob. Turn it up ($w=3$) and the input's effect triples. Turn it to zero and the input is ignored. Make it negative and the input flips sign. Every giant matrix later is just a grid of these knobs.

PICTURE. Look at the figure: the horizontal axis is the input $x$, the vertical axis is the output $w\cdot x$. The line's steepness is $w$. Three lines are drawn — a gentle one ($w=0.5$), a steep one ($w=2$), and a downhill one ($w=-1$).

Notice the flaw already: every line goes through the origin, and every line is straight. No matter what knob we turn, $x=0$ always gives $0$, and the relationship is always a straight line. That straightness is the enemy we fight in Step 4.

Step 2 — Adding a bias: sliding the line off the origin

WHAT. We add one more number, $b$ (the "bias"), after the multiply: the machine now computes $w\cdot x+b$.

WHY. In Step 1 every line was chained to the origin. But real features don't switch on at exactly zero. Maybe "this pattern is present" should trigger when $x>3$, not $x>0$. The bias is a shift — it slides the whole line up or down so the machine can choose where the interesting threshold sits. The symbol $b$ appears added, on its own, because it must act regardless of the input's size.

PICTURE. Same axes. One line is $w\cdot x$ (through origin, dashed). The solid amber line is $w\cdot x+b$ — identical steepness, but lifted up by exactly $b$. The vertical amber bracket marks the lift.

Reveal check:

Step 3 — Many inputs → the matrix row is a dot product

WHAT. A token isn't one number; it is a list of $d_{\text{model}}$ numbers, $x=[x_1,x_2,\dots ,x_d]$ (its "embedding"). To make one output number we give each input its own weight and add them all up: $z=w_1{x}_1+w_2{x}_2+\cdots +w_d{x}_d+b$

WHY. One knob can only react to one number. But a useful feature — say "positive sentiment AND past tense" — depends on several input dimensions at once. So we need one knob per input dimension. This weighted sum is called a dot product. We use a dot product specifically because it answers the question "how much does this input pattern match my stored pattern of weights?" — big when the input lines up with the weights, small or negative when it doesn't. No other single operation both mixes all inputs and stays linear.

PICTURE. The figure shows the input as a vertical stack of bars (heights $x_1\dots x_4$) and the weights as a second stack. Each pair is multiplied (amber links), the products flow into a summing node, and the bias drops in at the end to give one output number $z$.

Now stack many such neurons. If we want $d_{ff}$ output numbers, we need $d_{ff}$ rows of weights. Bundling all rows into one grid gives the matrix $W_1\in {\mathbb{R}}^{d_{\text{model}}\times d_{ff}}$, and all the biases into the vector $b_1$. That single line $x{W}_1+b_1$ is nothing more than "run Step 3's dot product $d_{ff}$ times in parallel." The matrix is not new magic — it is a rack of Step-1 knobs.

Step 4 — Why it collapses without a bend: the non-linearity

WHAT. We apply $\text{ReLU}(z)=\max (0,z)$ to every neuron's output: keep it if positive, clamp it to zero if negative.

WHY. This is the heart of the whole page. Suppose we did NOT bend — we just did matrix, then matrix: $x{W}_1{W}_2$. Multiplying two matrices gives another matrix $W_3=W_1{W}_2$. So two straight layers = one straight layer. Stacking 24 of them = still one straight line's worth of power! The parent note's warning — "stacking would collapse into a single linear transformation" — is exactly this. We need a kink, an operation that is not a straight line, sitting between the two matrices. ReLU is the simplest kink: flat for negatives, sloped for positives.

PICTURE (left). The ReLU graph: a flat floor at zero for $x<0$, then a $45°$ ramp for $x>0$. The corner at the origin is the "bend" — mark it in amber.

PICTURE (right). The proof of collapse: two straight lines composed give one straight line (still straight); but a straight line through a ReLU gives a genuine elbow — a shape no single matrix could make.

Step 5 — Expand then contract: the $4\times$ funnel

WHAT. The first matrix grows the vector from $d_{\text{model}}$ up to $d_{ff}=4d_{\text{model}}$; ReLU bends it; the second matrix $W_2$ shrinks it back to $d_{\text{model}}$.

WHY. Why go up first? Each hidden neuron is one detector (Step 3). With only $d_{\text{model}}$ neurons you get $d_{\text{model}}$ detectors — a bottleneck. Widening to $4\times$ gives four times as many detectors, so the layer can look for many more distinct patterns at once. Then why come back down? Because the output must be added to the residual stream, which lives in $d_{\text{model}}$ dimensions. The second matrix is forced to summarise the $d_{ff}$ raw detections into $d_{\text{model}}$ useful numbers — a compression that keeps only what matters.

PICTURE. A funnel drawn as three vertical columns of nodes: a narrow input column ($d_{\text{model}}$), a wide hidden column ($d_{ff}=4d_{\text{model}}$) with amber ReLU'd nodes, and a narrow output column back to $d_{\text{model}}$. Arrows fan out (expansion $W_1$) then fan in (contraction $W_2$).

Step 6 — Position-wise: the same funnel, stamped on every token

WHAT. Given a whole sentence $X=[x_1,\dots ,x_n]$, the FFN applies the identical funnel to each token separately: $\text{FFN}(X)=[\text{FFN}(x_1),\text{FFN}(x_2),\dots ,\text{FFN}(x_n)]$

WHY. By this point the attention sublayer has already mixed information between tokens, so each $x_i$ already "knows" about its neighbours. The funnel's job is pure per-token feature extraction; it does not need to peek sideways. Same weights everywhere means one funnel is reused $n$ times — this is why it parallelises perfectly and why it is literally a $1\times 1$ convolution.

PICTURE. Six token slots ("The cat sat on the mat") each feeding an identical grey funnel; a single shared weight-box connects to all of them with dotted lines to stress "same weights, different inputs." No horizontal arrows between funnels — that absence is the point.

Step 7 — Degenerate cases: check the machine never breaks

WHAT. Push weird inputs through and confirm each stays sane.

WHY. A trustworthy derivation must survive its own edge cases. Four to check:

1. All-negative pre-activation. If every $x{W}_1+b_1$ entry is negative, ReLU zeroes the entire hidden vector: $h=0$. Then $\text{FFN}(x)=0\cdot W_2+b_2=b_2$. The layer outputs just its bias — a constant. Not broken, just uninformative; the residual connection still carries $x$ forward, so no signal is lost.
2. Zero input, $x=0$. Then $x{W}_1+b_1=b_1$; the output depends only on the biases. The line-through-origin problem of Step 1 is exactly why $b_1$ had to exist.
3. Huge positive input. ReLU is unbounded, so $h$ scales linearly — but the next LayerNorm rescales it, so no explosion reaches the next block.
4. Modern smooth activations. Swap ReLU for GELU or SiLU: the hard corner becomes a soft curve, so tiny negatives leak a small gradient instead of dying. Everything above still holds; only the shape of the bend changes.

PICTURE. Four mini-panels, one per case: the all-negative flatline to $b_2$; the $x=0$ landing on $b_1$; the unbounded ramp tamed by a LayerNorm bracket; and ReLU's sharp corner overlaid with GELU's smooth curve.

The one-picture summary

Everything collapses into a single diagram: one token vector enters (Step 1–3 build each connection), $W_1$ fans it out (Step 5), ReLU bends it (Step 4), $W_2$ fans it back in, the bias shifts it (Step 2), and the whole stamp is reused per position (Step 6) with edge behaviour noted (Step 7).

Related reading: Universal Approximation Theorem (why two layers can learn anything), Batch Normalization vs Layer Normalization, Positional Encoding, and the full Transformer Encoder Block.