Foundations — Layer normalization

Before you can read the parent note, you must own every symbol it throws at you. We build them one at a time, each anchored to a picture, each earned before the next uses it. Nothing here assumes you have seen the notation before.

1. A vector — the list of numbers

The parent writes $\mathbf{x}=(x_1,\dots ,x_H)$. Let us un-pack every piece.

The picture. Think of a row of bars, one bar per number, its height equal to that number. That row is the vector.

Why the topic needs it. A neuron does not emit a single number — it emits a whole row of them, one per "unit". LayerNorm works on that entire row at once, so we need a name for the row (that is $\mathbf{x}$) and a name for each individual bar (that is $x_i$).

2. $H$ — how long the list is

The picture. Count the bars in the previous figure. That count is $H$.

Why the topic needs it. LayerNorm averages across the features. To average, you must divide by how many things you added up — that count is $H$. Every formula that begins $\frac{1}{H}\sum$ is saying "add these up and divide by how many there are", i.e. take an average.

3. $\sum$ — the adding-up machine

The formulas use $\sum _{i=1}^H{x}_i$. This scares people; it should not.

The picture. Imagine sliding a bucket from the first bar to the last, tipping each bar's height into the bucket. The final level in the bucket is the sum.

Why the topic needs it. Every statistic in LayerNorm — the mean, the variance — is "add something over all features". The $\sum$ is that adding.

4. $\mu$ — the mean (the balance point)

Now we can read the first real formula: $\mu =\frac{1}{H}\sum _{i=1}^H{x}_i$.

The picture. Put the bars on a see-saw. The mean is the point where the see-saw balances. Numbers above the mean pull right; numbers below pull left; at $\mu$ the pulls cancel.

Why the topic needs it. LayerNorm's first job is to recentre the row so its balance point sits at zero. To move it to zero you must first know where it currently balances — that is $\mu$.

5. Deviation $x_i-\mu$ — distance from the balance point

The picture. Draw a horizontal line at height $\mu$. Each deviation is the signed gap from that line up (or down) to the top of a bar.

Why the topic needs it. "Centring to zero" is replacing each $x_i$ with its deviation. A remarkable fact used later: the deviations always add to zero, because the amounts above the balance point exactly match the amounts below.

6. ${\sigma }^2$ — the variance (how spread out)

Next: ${\sigma }^2=\frac{1}{H}\sum _{i=1}^H(x_i-\mu {)}^2$.

The picture. A narrow cluster of bars near the mean → small $\sigma$. A wide fan of bars → large $\sigma$.

Why the topic needs it. After centring, LayerNorm rescales the row so its spread becomes exactly 1. To shrink or stretch the spread to 1, you must first measure it — that is $\sigma$.

7. ${\hat{x}}_i$, the hat, and $ϵ$ — the standardized value

Now the payoff line: ${\hat{x}}_i=\frac{x_i-\mu }{\sqrt{{\sigma }^2+ϵ}}$.

Why the topic needs it. This one formula is the heart of LayerNorm. Everything before it (vector, $H$, $\sum$, $\mu$, deviation, $\sigma$) existed only so this line could be read and believed.

8. ${\gamma }_i$, ${\beta }_i$ — the two learnable knobs

Final line: $y_i={\gamma }_i{\hat{x}}_i+{\beta }_i$.

How these feed the topic

Read top-to-bottom: the vector and its length give us the mean; the mean gives deviations; deviations give variance and spread; spread plus $ϵ$ let us standardize; and the two knobs finish the job. Only once all of these are in hand does the parent topic Layer normalization make sense.

Where these ideas reappear

• The same mean/variance machinery, but averaged down a column instead of across a row, is Batch normalization — its column-vs-row difference is the whole story.
• Normalizing to keep activations tidy is a direct defence against Vanishing and exploding gradients.
• The instability LayerNorm fights — shifting input distributions — is Internal covariate shift.
• LayerNorm is the normalizer of choice inside Transformers, works alongside Residual connections, and has a leaner cousin RMSNorm that drops the mean-subtraction step.
• The "tidy range" matters because the next stage is usually an activation function — see Activation functions.
• Prefer Hindi? Yeh Hinglish mein padho →

Equipment checklist

Self-test: can you answer each without peeking?