Visual walkthrough — Layer normalization

We work with one example: a single list of numbers coming out of a layer. Call that list $\mathbf{x}=(x_1,x_2,\dots ,x_H).$

Reading the symbols:

• $\mathbf{x}$ (bold) — the whole list, one example's worth of numbers.
• $x_i$ — the $i$-th number in that list. The little $i$ is just a counter: $x_1$ is the first, $x_2$ the second.
• $H$ — how many numbers are in the list (the number of features / units). In our running example $H=4$.

That's all the notation you need to start. Everything else we build.

The page follows four moves, kept as separate numbered steps so you always know when $ϵ$ enters: Centre (Step 3), Squeeze (Step 5), Guard with $ϵ$ (Step 6), Restyle (Step 9).

Step 1 — Lay the raw values on a number line

WHAT. Take one example $\mathbf{x}=(1,2,3,4)$ and simply plot each value as a dot on a horizontal axis.

WHY. Before we do any arithmetic, we must see the problem: these four numbers sit off to one side (all positive, centred near 2.5) and are spread out by a certain amount. LayerNorm's whole job is to slide this cluster to sit on zero and squeeze it to a standard width. You can't appreciate a fix you can't picture.

PICTURE. The four lavender dots below are the raw $x_i$. Notice they are not centred on 0 — their middle is off to the right.

Step 2 — Find the centre: the mean $\mu$

WHAT. Add all the numbers and divide by how many there are.

$\mu =\underset{\text{divide by count}}{\underbrace{\frac{1}{H}}}\underset{\text{add them all up}}{\underbrace{\sum _{i=1}^H{x}_i}}$

Symbol by symbol:

• ${\sum }_{i=1}^H{x}_i$ — the big Greek "S" ($\Sigma$, sigma) means sum. Start at $i=1$, go to $i=H$, add every $x_i$: here $1+2+3+4=10$.
• $\frac{1}{H}$ — divide that total by the count $H=4$, giving $\mu =10/4=2.5$.
• $\mu$ (Greek "mu") — the balance point of the dots, the value they average to.

WHY. To recentre the cluster we first need to know where its centre is. The mean is exactly "the place where the dots balance." Nothing fancier will do: it is the unique number whose deviations sum to zero (we use that fact in Step 4).

PICTURE. The coral line marks $\mu =2.5$ — it sits right in the middle of the lavender dots, like the pivot of a see-saw.

Step 3 — Slide everything to zero (subtract $\mu$)

WHAT. Replace each $x_i$ by its deviation from the mean: $d_i=x_i-\mu .$

For $\mathbf{x}=(1,2,3,4)$, $\mu =2.5$: $(d_1,d_2,d_3,d_4)=(-1.5,-0.5,0.5,1.5).$

• $x_i-\mu$ — "how far is this dot from the centre, and on which side?" Negative = left of centre, positive = right.

WHY. We want the finished output to have mean zero (the parent note's Step 1 goal: "so the layer isn't biased high or low"). Subtracting the same number $\mu$ from all of them slides the whole cluster left until its centre lands on 0 — it changes position but not spread. This is the Centre move.

PICTURE. The cluster has shifted so the coral line (now at 0) passes through its middle. The distances between dots are unchanged — only the whole group moved.

Step 4 — Measure the spread: variance ${\sigma }^2$ and standard deviation $\sigma$

WHAT. Square each deviation, then average the squares: ${\sigma }^2=\frac{1}{H}{\sum }_{i=1}^H\underset{\text{squared distance from centre}}{\underbrace{(x_i-\mu {)}^2}}$ and define the standard deviation as its square root: $\sigma =\sqrt{{\sigma }^2}.$

For our deviations $(-1.5,-0.5,0.5,1.5)$:

• squares: $(2.25,0.25,0.25,2.25)$,
• average: $\frac{2.25+0.25+0.25+2.25}{4}=\frac{5}{4}=1.25={\sigma }^2$,
• root: $\sigma =\sqrt{1.25}\approx 1.118$.

Symbol by symbol:

• $(x_i-\mu {)}^2$ — squaring does two jobs: it kills the sign (left and right both count as "far"), and it punishes far dots more than near ones.
• ${\sigma }^2$ (Greek "sigma", squared) — the average squared distance; a single number for "how wide is this cluster."
• $\sigma =\sqrt{{\sigma }^2}$ — the standard deviation: the width put back into the same units as the data (here $\approx 1.118$). This is the number we will divide by in Step 5.

WHY squaring and not, say, absolute value? Squared distance is the measure that makes the rescaling in Step 5 give exactly variance 1 (we prove this in Step 7). It's the natural partner of the mean.

PICTURE. Each dot's distance to the centre is drawn as a bar; its squared length is the shaded square. $\sigma$ (mint) is the "typical" half-width of the cluster.

Step 5 — Squeeze to standard width: divide by $\sigma$

WHAT. Divide every centred value by the standard deviation. We first write the ideal squeeze (no safety term yet), because it's the cleanest to picture: ${\hat{x}}_i^{\text{ideal}}=\frac{x_i-\mu }{\sigma }=\frac{x_i-\mu }{\sqrt{{\sigma }^2}}$

With $\sigma =1.118$: $\hat{\mathbf{x}}=\frac{(-1.5,-0.5,0.5,1.5)}{1.118}=(-1.342,-0.447,0.447,1.342).$

Symbol by symbol:

• numerator $x_i-\mu$ — the already-centred value from Step 3.
• $\sigma =\sqrt{{\sigma }^2}$ — the cluster's width from Step 4. Dividing by it rescales the whole cluster to a standard width of 1.
• ${\hat{x}}_i$ ("x-hat") — the standardized value: mean 0, variance 1.

WHY. Step 3 fixed the position; this Squeeze fixes the size. After both, every example's row looks the same shape regardless of how big or lopsided it started — that stability is exactly what the next layer wants. (Step 6 will patch the denominator so a flat row can't break this.)

PICTURE. The wide green cluster (before) is squeezed to the narrow one (after), now spanning a standard width. Same shape, standardized scale.

Step 6 — Guard the denominator: add $ϵ$

WHAT. Replace the bare width $\sigma =\sqrt{{\sigma }^2}$ by a guarded width $\sqrt{{\sigma }^2+ϵ}$, giving the real LayerNorm formula: ${\hat{x}}_i=\frac{x_i-\mu }{\sqrt{{\sigma }^2+ϵ}}$

Symbol by symbol:

• $ϵ$ (Greek "epsilon", e.g. $10^{-5}$) — a tiny positive number added inside the square root before taking the root. If a row were nearly constant, ${\sigma }^2\approx 0$ and Step 5 would divide by ~0 → NaN. Adding $ϵ$ keeps the denominator safely above zero.

WHY. This is the Guard move — the single reason the formula in Step 5 is production-safe. It costs almost nothing on normal rows and rescues the degenerate ones (Step 8).

PICTURE. Same squeezed cluster as Step 5, but the width bar is now $\sqrt{{\sigma }^2+ϵ}$ — a whisker wider than $\sigma$, so the dots sit a whisker inside width 1.

Step 7 — Prove the ideal output is mean 0, variance 1

WHAT. Check the two guarantees on the standardized row, in the ideal case $ϵ=0$ (Step 6 already showed what $ϵ$ changes).

Mean of ${\hat{x}}_i$: $\frac{1}{H}{\sum }_i{\hat{x}}_i=\frac{1}{H}{\sum }_i\frac{x_i-\mu }{\sqrt{{\sigma }^2}}=\frac{\stackrel{=H\mu }{\overbrace{(\sum _i{x}_i)}}-H\mu }{H\sqrt{{\sigma }^2}}=0.$

Variance of ${\hat{x}}_i$ (its values are already centred, so variance = mean of squares): $\frac{1}{H}{\sum }_i{\hat{x}}_i^2=\frac{1}{H}{\sum }_i\frac{(x_i-\mu {)}^2}{{\sigma }^2}=\frac{1}{{\sigma }^2}\cdot \underset{={\sigma }^2}{\underbrace{\frac{1}{H}\sum _i(x_i-\mu {)}^2}}=1.$

WHY. This is the payoff: with $ϵ=0$, not "roughly" but exactly mean 0 and variance 1. This is why we chose squared distance in Step 4 — it makes the second line collapse to ${\sigma }^2/{\sigma }^2=1$. Add the real $ϵ$ and you get ${\sigma }^2/({\sigma }^2+ϵ)$ instead, as Step 6 showed.

PICTURE. Numerically: our $\hat{\mathbf{x}}=(-1.342,-0.447,0.447,1.342)$ sums to $0$ and its mean square is $\frac{1.8+0.2+0.2+1.8}{4}=1$. The green bars land symmetrically about 0 with the mint width equal to 1.

Step 8 — The degenerate case: a flat row

WHAT. What if every feature is the same, e.g. $\mathbf{x}=(5,5,5,5)$?

• $\mu =5$, every deviation $=0$, so ${\sigma }^2=0$.
• Without $ϵ$: ${\hat{x}}_i=\frac{0}{\sqrt{0}}=\frac{0}{0}$ → undefined / NaN.
• With $ϵ=10^{-5}$ (the Step 6 guard): ${\hat{x}}_i=\frac{0}{\sqrt{0+10^{-5}}}=\frac{0}{0.00316}=0$. Safe.

WHY. A constant row has no spread to standardize — there's genuinely nothing to normalize, and the honest answer is "all zeros." The $ϵ$ lets the formula reach that answer instead of exploding. This is not a corner you can skip: near-constant rows really occur (dead activations, padding tokens).

PICTURE. Left panel: all dots stacked at one value, coral mean line runs right through them, width $=0$. Right panel: with $ϵ$, the output collapses cleanly to $0$ instead of blowing up.

Step 9 — Restore freedom: the learnable knobs $\gamma ,\beta$

WHAT. Give each feature its own scale ${\gamma }_i$ and shift ${\beta }_i$: $y_i=\underset{\text{stretch/shrink}}{\underbrace{{\gamma }_i}}{\hat{x}}_i+\underset{\text{slide}}{\underbrace{{\beta }_i}}$

With $\gamma =(2,2,2,2)$, $\beta =(1,1,1,1)$ on $\hat{\mathbf{x}}=(-1.342,-0.447,0.447,1.342)$: $\mathbf{y}=2\hat{\mathbf{x}}+1=(-1.684,0.106,1.894,3.684).$

Symbol by symbol:

• ${\gamma }_i$ (Greek "gamma") — multiplies ${\hat{x}}_i$: a new width knob (here ×2 makes the cluster twice as wide).
• ${\beta }_i$ (Greek "beta") — added on: a new centre knob (here +1 slides the cluster to mean 1).
• $y_i$ — the value that actually leaves LayerNorm and enters the next layer.

WHY. Mean 0 / variance 1 is a good starting point, but sometimes a bad ending point — e.g. the next activation function (like tanh) may want a wider input range. This Restyle move lets the network choose the final scale and shift by gradient descent. Crucially, if not normalizing is best, the net can learn ${\gamma }_i=\sqrt{{\sigma }^2+ϵ}$ and ${\beta }_i=\mu$ to exactly undo Steps 3–6. So LayerNorm can never make things strictly worse.

PICTURE. The mint standardized cluster (mean 0, width 1) is stretched ×2 (butter) and slid +1 to become the final butter cluster centred at 1.

The one-picture summary

The whole pipeline is four moves on one row of numbers: Centre (subtract $\mu$, Step 3), Squeeze (divide by $\sigma$, Step 5), Guard (add $ϵ$ under the root, Step 6), Restyle (apply $\gamma ,\beta$, Step 9). Because every step uses only this row's statistics, LayerNorm never touches other examples — that's why it's identical at train and test time and works with a batch of size 1 (Transformers, RMSNorm, Residual connections all lean on this).

Flashcards