Before you can read the parent note, you must be fluent in a small pile of symbols. This page defines each one from absolute zero: plain words → the picture → why the topic needs it. Read top to bottom; every item leans on the one above it.
Picture a scalar as one dot on a ruler. A vector is several rulers side by side, each holding its own dot. The whole collection of dots is the vector.
Why the topic needs it: a Transformer never processes "a word." It processes the word's vector — its list of numbers. In the parent note, x=[1,0,−1,2] is exactly such a list.
Read Rd aloud as "the space of all lists of length d." The R is a fancy letter for "real numbers" (any number on the ruler, including decimals and negatives). The little superscript d is just the length.
Why the topic needs it: the FFN's entire job is described by how the dimension changes — dmodel→dff→dmodel. You cannot follow "expand then contract" without knowing what a dimension is.
The single most important operation in the whole topic is xW — feeding a vector into a matrix to get a new vector of possibly different length.
Why the topic needs it: xW1 (expansion) and hW2 (projection) are both this operation. The number of columns of W decides the output length — that is literally how the FFN changes dimension. If W1 has dff columns, the output has dff slots.
Picture xW as choosing a direction and steepness; the bias bslides the whole result along so the detector can fire at the right threshold. Without a bias, every detector would be forced to pass through zero.
Why the topic needs it: both FFN layers are xW+b. The bias is what lets a neuron say "fire only when the pattern is strongly present," not merely "present at all."
Here is the pivotal question the whole topic answers: why bother with two layers and a curve — why not one big matrix?
To make the network able to draw bent decision boundaries, we must insert something that is not a straight line between the two matrices. That "something" is an activation function.
Why this tool and not another: a matrix answers "which linear combination?"; only a curve can answer "should this feature turn on or stay off?" — a decision matrices structurally cannot make. See Activation Functions and Universal Approximation Theorem for why any smooth curve, given enough width, unlocks any continuous function.
The parent throws several curve-symbols at you. Here they are, each with its picture.
Recall Which curve is the hard hinge?
Which of the four has a sharp corner and a totally flat left side? ::: ReLU — max(0,x); the others are smooth everywhere.
Why the topic needs all four: the parent traces the evolution ReLU → GELU → SiLU. GELU is built from Φ (or its tanh stand-in); SiLU is built from σ. You cannot read that story without knowing each symbol's shape.
Picture a sentence as a row of boxes, one box per word, each box holding that word's vector.
Why the topic needs it: this single idea separates the FFN from an ordinary neural net. The mixing between boxes was already done earlier by attention; the FFN deliberately works one box at a time. This is also why it links to Residual Connections and Layer Normalization, which wrap the FFN box-by-box.
Why the topic needs them: the parent's final formula is LayerNorm(x+FFN(x)). Both pieces are context you must already own; contrast with Batch Normalization vs Layer Normalization for why it normalizes per-vector, not per-batch.
