Layer normalization: Worked examples — AI-ML deep dive

This page is a drill of cases. The Layer normalization parent note built the machine; here we run every kind of input through it so you never meet a surprise. We compute LayerNorm by hand for tiny vectors, then push it into the strange corners: a flat (constant) vector, a two-value vector, a single-feature row, negatives, huge scales, the $γ, β$ affine layer, and a word problem.

Everything uses only what the parent defined. As a one-line reminder, for one example's vector $x = (x_{1}, \dots, x_{H})$ :

The scenario matrix

Every input a LayerNorm layer can receive falls into one of these cells. The examples below are labelled with the cell(s) they cover, so by the end every row of this table has a worked case.

Cell	What makes it special	Where the danger is	Covered by
A. Ordinary spread	mixed positive values, $σ^{2} > 0$	none — the "textbook" run	Ex 1
B. Pure scaling	same shape as A but $\times k$	is it really scale-invariant?	Ex 2
C. Pure shift	same shape but $+ c$ to every entry	does adding a constant change $\overset{x}{^}$ ?	Ex 3
D. Negatives + asymmetry	some $x_{i} < 0$ , lopsided	sign bookkeeping in deviations	Ex 4
E. Two features, $H = 2$	smallest non-trivial row	$\overset{x}{^}$ collapses to $\pm 1$	Ex 5
F. Degenerate / constant	all $x_{i}$ equal, $σ^{2} = 0$	division by ~zero — the $ϵ$ case	Ex 6
F′. Single feature, $H = 1$	one number, $σ^{2} = 0$ always	same $ϵ$ rescue, output $0$	Ex 6b
G. Affine restore	apply $γ, β$ ; also "undo"	knobs re-shape the output	Ex 7
H. Word problem	real activations drift up 100 $\times$	why this stabilises the next layer	Ex 8
I. Exam twist	solve backwards for $γ, β$	invert the affine step	Ex 9

The examples

Worked example Example 1 — Cell A: an ordinary vector

$x = (2, 4, 4, 6)$ , $H = 4$ , $ϵ = 0$ , $γ = 1, β = 0$ . Find $\hat{x}$ .

Forecast: guess — will the answer be symmetric around zero? (The input is symmetric around $4$ , so… yes.)

Mean: $μ = \frac{2 + 4 + 4 + 6}{4} = 4$ . Why this step? We center the row so it isn't biased high or low — subtracting $μ$ is what makes the mean become $0$ .
Deviations: $(2 - 4, 4 - 4, 4 - 4, 6 - 4) = (- 2, 0, 0, 2)$ . Why this step? These are the raw distances from the center; standardizing just rescales them.
Variance: squares $(4, 0, 0, 4)$ , mean $= \frac{8}{4} = 2$ , so $σ^{2} = 2$ , $σ^{2} = 2 \approx 1.4142$ . Why this step? $σ$ is the "typical size" of a deviation; dividing by it makes the spread equal to $1$ .
Standardize: $\hat{x} = (\frac{- 2}{2}, 0, 0, \frac{2}{2}) = (- 1.4142, 0, 0, 1.4142)$ . Why this step? Dividing each centered value by $σ^{2}$ shrinks the big deviations and grows the small ones to a common scale, so the whole row ends with variance exactly $1$ — the controlled size the next layer expects.

Verify: sum $= 0$ ✓ (mean 0). Mean of squares $= \frac{2 + 0 + 0 + 2}{4} = 1$ ✓ (variance 1). Symmetric as forecast. ✓

Worked example Example 2 — Cell B: multiply everything by

$k = 10$ $x = (20, 40, 40, 60)$ (Example 1 $\times 10$ ). Predict $\hat{x}$ before computing.

Forecast: the shape is identical, just $10 \times$ bigger. If LayerNorm cares only about shape, the answer should be the same as Ex 1.

Mean: $μ = 40 = 10 \times 4$ . Why this step? Scaling every entry by $k$ scales the mean by $k$ — the center moves with the data.
Deviations: $(- 20, 0, 0, 20) = 10 \times (- 2, 0, 0, 2)$ . Why this step? Deviations scale by $k$ too.
Variance: $σ^{2} = \frac{400 + 0 + 0 + 400}{4} = 200 = 1 0^{2} \times 2$ , so $σ^{2} = 102$ . Why this step? Variance is a square, so it scales by $k^{2}$ and its square root by $k$ .
Standardize: each deviation $\div σ^{2}$ : the two factors of $10$ cancel, giving $(- 1.4142, 0, 0, 1.4142)$ . Why this step? This cancellation is the whole point: $\frac{k \cdot ( x _{i} - μ )}{k \cdot σ} = \frac{x _{i} - μ}{σ}$ .

Verify: identical to Ex 1 ✓. LayerNorm is scale-invariant (before $γ, β$ ).

Worked example Example 3 — Cell C: add a constant

$c = 100$ to every entry $x = (102, 104, 104, 106)$ (Example 1 $+ 100$ ). Predict $\hat{x}$ .

Forecast: we only slid the whole row rightward. Deviations from the mean shouldn't move — expect Ex 1's answer again.

Mean: $μ = 104 = 4 + 100$ . Why this step? Adding $c$ to every entry adds $c$ to the mean.
Deviations: $(102 - 104, \dots) = (- 2, 0, 0, 2)$ — unchanged from Ex 1. Why this step? The $+ 100$ appears in both $x_{i}$ and $μ$ , so it cancels in $x_{i} - μ$ .
Variance: $σ^{2} = 2$ (deviations unchanged $\Rightarrow$ variance unchanged).
Standardize: $(- 1.4142, 0, 0, 1.4142)$ . Why this step? Both the centered values and $σ^{2}$ are identical to Ex 1, so the ratio — and therefore $\hat{x}$ — must be identical too; the constant slid away completely.

Verify: identical to Ex 1 ✓. So LayerNorm is shift-invariant too. Together with Ex 2: any input of the form $k x_{Ex1} + c$ produces the same $\hat{x}$ .

Examples 1–3 are three different raw rows that collapse onto one normalized answer. The figure below plots all three raw inputs (very different heights, dashed) and the single orange curve they all become after normalizing — a picture of scale- and shift-invariance. If the image fails to load: the dashed teal/plum/tan lines sit at heights around $2$ – $6$ , $20$ – $60$ , and $102$ – $106$ respectively, while one thick orange line hits exactly $(- 1.41, 0, 0, + 1.41)$ for all three.

Figure s01 — Three raw rows (dashed, wildly different heights) all normalize to the one orange curve $(- 1.41, 0, 0, 1.41)$ : LayerNorm keeps only the shape.

Worked example Example 4 — Cell D: negatives and lopsided spread

$x = (- 4, - 2, 0, 10)$ , $H = 4$ , $ϵ = 0$ . Find $\hat{x}$ .

Forecast: one entry ( $10$ ) is far above the rest, so its $\overset{x}{^}$ should be a large positive number; the three low ones should be modest negatives.

Mean: $μ = \frac{- 4 - 2 + 0 + 10}{4} = \frac{4}{4} = 1$ . Why this step? Even with negatives, the mean is just the balance point of the row.
Deviations: $(- 4 - 1, - 2 - 1, 0 - 1, 10 - 1) = (- 5, - 3, - 1, 9)$ . Why this step? Sign matters: entries below the mean give negative deviations. Sum $= - 5 - 3 - 1 + 9 = 0$ ✓ (deviations always sum to zero — a good running check).
Variance: squares $(25, 9, 1, 81)$ , sum $= 116$ , $σ^{2} = \frac{116}{4} = 29$ , $29 \approx 5.3852$ . Why this step? Squaring removes the signs so the spread is a positive size.
Standardize: $\hat{x} = (\frac{- 5}{5.3852}, \frac{- 3}{5.3852}, \frac{- 1}{5.3852}, \frac{9}{5.3852}) \approx (- 0.9285, - 0.5571, - 0.1857, 1.6713)$ . Why this step? Dividing every centered value by the same $σ^{2}$ keeps the signs (negatives stay negative) while forcing the overall spread to $1$ ; that is why the lone big deviation $9$ ends up as the single large positive output.

Verify: sum $\approx 0$ ✓. Mean of squares: $\frac{0.8621 + 0.3103 + 0.0345 + 2.7931}{4} \approx 1$ ✓. The outlier $10$ became the big $+ 1.6713$ , exactly as forecast. ✓

Worked example Example 5 — Cell E: the smallest row,

$H = 2$ $x = (a, b)$ with $a \neq = b$ , $ϵ = 0$ . What is $\hat{x}$ in general?

Forecast: with only two numbers, once you force mean 0 and variance 1 there's no freedom left — the two outputs must be fixed, probably $\pm 1$ .

Mean: $μ = \frac{a + b}{2}$ . Why this step? The mean of two numbers is their midpoint; centering will push that midpoint to $0$ , so each entry ends up equidistant from $0$ .
Deviations: $a - μ = \frac{a - b}{2}$ and $b - μ = \frac{b - a}{2} = - \frac{a - b}{2}$ . Why this step? For $H = 2$ the two deviations are exact negatives — the row is automatically symmetric about its mean.
Variance: $σ^{2} = \frac{1}{2} [(\frac{a - b}{2})^{2} + (\frac{a - b}{2})^{2}] = (\frac{a - b}{2})^{2}$ , so $σ^{2} = \frac{a - b}{2}$ . Why this step? Both squared deviations are equal, so the variance is just one of them.
Standardize: $\overset{a}{^} = \frac{( a - b ) /2}{∣ ( a - b ) /2∣} = sign (a - b)$ , and $\hat{b} = - \overset{a}{^}$ . Why this step? Dividing a number by its own absolute value leaves only its sign (the fact from the definition box above): $\frac{t}{∣ t ∣} = sign (t)$ .

Conclusion: for any $H = 2$ input with $a \neq = b$ , $\hat{x} = (+ 1, - 1)$ if $a > b$ , else $(- 1, + 1)$ . Concretely $x = (7, 3) \Rightarrow \hat{x} = (1, - 1)$ ; $x = (3, 7) \Rightarrow (- 1, 1)$ .

Verify: sum $= 0$ ✓, mean square $= \frac{1 + 1}{2} = 1$ ✓. A length-2 row always normalizes to $\pm 1$ regardless of the actual values — it keeps only the ordering.

Worked example Example 6 — Cell F: the degenerate constant row (the

$ϵ$ case) $x = (5, 5, 5, 5)$ . First set $ϵ = 0$ , then use $ϵ = 1 0^{- 5}$ .

Forecast: every entry equals the mean, so every deviation is $0$ . With $ϵ = 0$ we'd compute $0/0$ = disaster. With $ϵ > 0$ the outputs should be a safe $0$ .

Mean: $μ = 5$ .
Deviations: $(0, 0, 0, 0)$ . Why this step? A flat row has no spread.
Variance: $σ^{2} = 0$ . Why this step? The average of the squared deviations is the average of four zeros — the spread of a constant row is exactly zero, which is precisely what makes the next step dangerous.
Standardize, $ϵ = 0$ : $\overset{x}{^}_{i} = \frac{0}{0} = \frac{0}{0}$ — undefined / NaN. Why this step? This is the exact failure the parent warned about — a near-constant row divides by ~zero and produces NaN, which then poisons every downstream gradient.
Standardize, $ϵ = 1 0^{- 5}$ : $\overset{x}{^}_{i} = \frac{0}{0 + 1 0 ^{- 5}} = \frac{0}{0.003162} = 0$ . Why this step? The numerator is genuinely $0$ (the value equals the mean), and now the denominator is a real positive number, so the answer is a clean $0$ — the row simply contributes nothing, which is correct.

Verify: with $ϵ > 0$ we get $\hat{x} = (0, 0, 0, 0)$ , finite and safe ✓. This is why $ϵ$ is not optional.

The figure below shows the output for the constant row as $ϵ$ sweeps from tiny to large: the teal line stays pinned at $0$ for every $ϵ > 0$ , and the orange marker shows the $ϵ \to 0$ cliff where it becomes $0/0$ . If the image fails to load: a flat teal line sits at height $0$ across the whole log-scale $ϵ$ axis; a dashed orange line at the far-left ( $ϵ \to 0$ ) marks the NaN cliff, and a dotted plum line at $ϵ = 1 0^{- 5}$ marks the safe typical value.

Figure s02 — For the constant row $(5, 5, 5, 5)$ , the output is a safe $0$ for every $ϵ > 0$ (teal); only $ϵ \to 0$ (orange dashed) gives the $0/0$ NaN cliff.

Worked example Example 6b — Cell F′: the single-feature row,

$H = 1$ $x = (7)$ — just one number. First $ϵ = 0$ , then $ϵ = 1 0^{- 5}$ . What is $\hat{x}$ ?

Forecast: with only one value, its mean is that value, so the deviation is $0$ and there is no spread at all — this should behave exactly like the constant row of Ex 6.

Mean: $μ = \frac{7}{1} = 7$ . Why this step? The mean of a single number is the number itself.
Deviation: $7 - 7 = 0$ . Why this step? A one-element row is always equal to its own mean, so its single deviation is forced to $0$ for any input value.
Variance: $σ^{2} = \frac{0 ^{2}}{1} = 0$ . Why this step? $H = 1$ makes $σ^{2} = 0$ unavoidably — a single-feature row is the permanent version of the degenerate case, not an accident.
Standardize, $ϵ = 0$ : $\frac{0}{0} = \frac{0}{0}$ — NaN, same cliff as Ex 6.
Standardize, $ϵ = 1 0^{- 5}$ : $\frac{0}{1 0 ^{- 5}} = 0$ . Why this step? Identical rescue to the constant row: the $ϵ$ keeps the denominator positive and the output is a safe $0$ .

Verify: $\hat{x} = (0)$ with $ϵ > 0$ ✓. Lesson: LayerNorm on a single feature can only ever output $γ \cdot 0 + β = β$ — it throws the value away and returns the learned shift. That is why LayerNorm is applied over a layer of many units, never over $H = 1$ .

Worked example Example 7 — Cell G: apply the affine layer, then undo it

Start from Ex 1's $\hat{x} = (- 1.4142, 0, 0, 1.4142)$ . (a) Apply $γ = (3, 3, 3, 3)$ , $β = (2, 2, 2, 2)$ . (b) Then find the $γ, β$ that would fully recover the original $x = (2, 4, 4, 6)$ .

Forecast (a): $y = 3 \overset{x}{^} + 2$ should have mean $\approx 2$ (the $β$ ) and standard deviation $\approx 3$ (the $γ$ ).

Compute (a): $y_{i} = 3 \overset{x}{^}_{i} + 2$ : $y = (3 (- 1.4142) + 2, 2, 2, 3 (1.4142) + 2) = (- 2.2426, 2, 2, 6.2426)$ . Why this step? $γ$ stretches the spread from $1$ to $3$ ; $β$ slides the center from $0$ to $2$ .
Check the forecast: mean $= \frac{- 2.2426 + 2 + 2 + 6.2426}{4} = 2$ ✓; the two nonzero $\overset{x}{^}$ entries were $\pm 1.4142$ , and multiplying by $γ = 3$ makes them sit $\pm 3 \times 1.4142 = \pm 4.2426$ away from the new center $2$ , i.e. at $- 2.2426$ and $6.2426$ — a spread exactly $3 \times$ Ex 1's. ✓ Why this step? Confirming mean $= β$ and spread $= γ \times$ (old spread) checks that $γ, β$ did exactly what we claimed and nothing else leaked in.
Recover the original (b): the parent note says the network can undo LayerNorm by choosing $γ_{i} = σ^{2} + ϵ$ and $β_{i} = μ$ . From Ex 1, $σ^{2} = 2 \approx 1.4142$ and $μ = 4$ . Why this step? $γ \overset{x}{^} + β = σ^{2} \cdot \frac{x - μ}{σ ^{2}} + μ = x$ — the standardization exactly cancels.
Confirm: $y_{i} = 1.4142 \overset{x}{^}_{i} + 4$ : $(1.4142 (- 1.4142) + 4, 4, 4, 1.4142 (1.4142) + 4) = (2, 4, 4, 6) = x$ ✓.

Verify: (a) gives $(- 2.2426, 2, 2, 6.2426)$ ; (b) reproduces $(2, 4, 4, 6)$ ✓. So LayerNorm never removes information — the affine knobs can restore it.

Worked example Example 8 — Cell H: word problem, activations blow up

An earlier layer usually outputs a token's feature row around $x = (1, 3, 5, 7)$ . During training its weights explode and it now emits $100 \times$ that: $(100, 300, 500, 700)$ . The next layer expects "normal-sized" inputs. Show LayerNorm shields it.

Forecast: by the scale-invariance we proved in Ex 2, the $100 \times$ blow-up should vanish after normalization — the next layer sees the same thing either way.

Normalize the calm row $(1, 3, 5, 7)$ : $μ = 4$ , deviations $(- 3, - 1, 1, 3)$ , squares $(9, 1, 1, 9)$ , $σ^{2} = 5$ , $5 \approx 2.2361$ . $\hat{x} = (- 1.3416, - 0.4472, 0.4472, 1.3416)$ . Why this step? Establish the baseline the next layer is used to.
Normalize the exploded row $(100, 300, 500, 700)$ : $μ = 400$ , deviations $100 \times (- 3, - 1, 1, 3)$ , $σ^{2} = 10 0^{2} \times 5$ , $σ^{2} = 1005$ . Each deviation over $σ^{2}$ : the $100$ s cancel $\Rightarrow \hat{x} = (- 1.3416, - 0.4472, 0.4472, 1.3416)$ . Why this step? Same cancellation as Ex 2 — the scale factor never survives standardization.
Compare: the two normalized rows are identical. Why this step? This is the stability promise: an upstream explosion in magnitude cannot disturb the downstream layer's input distribution.

Verify: both rows normalize to $(- 1.3416, - 0.4472, 0.4472, 1.3416)$ ✓, sum $0$ , variance $1$ . The next layer is fully shielded from the $100 \times$ drift. ✓

Worked example Example 9 — Cell I: exam twist, solve backwards

A LayerNorm layer received a row whose standardized values were $\hat{x} = (- 1, - 1, 1, 1)$ (so $H = 4$ ). Its output was $y = (0, 0, 4, 4)$ . All $γ_{i}$ are equal to one number $g$ and all $β_{i}$ to one number $b$ . Find $g$ and $b$ .

Forecast: the output has mean $2$ and the entries sit $\pm 2$ from it, while $\overset{x}{^}$ sits $\pm 1$ . So the stretch is $2$ and the shift is $2$ : expect $g = 2, b = 2$ .

Write the affine equation for one known entry. For the first: $y_{1} = g \overset{x}{^}_{1} + b \Rightarrow 0 = g (- 1) + b$ . Why this step? $y = γ \overset{x}{^} + β$ is linear, so two data points pin down the two unknowns.
Second independent entry: the third: $4 = g (1) + b$ . Why this step? We need a second equation with a different $\overset{x}{^}$ value ( $+ 1$ vs $- 1$ ) to separate $g$ from $b$ .
Solve the pair. Add them: $0 + 4 = (- g + b) + (g + b) = 2 b \Rightarrow b = 2$ . Substitute into $4 = g + b$ : $g = 2$ . Why this step? Adding cancels $g$ and isolates $b$ ; back-substitution gives $g$ .

Verify: plug back: $y = 2 \overset{x}{^} + 2$ gives $(2 (- 1) + 2, - 2 + 2, 2 (1) + 2, 2 (1) + 2) = (0, 0, 4, 4) = y$ ✓. So $g = 2, b = 2$ . ✓

The 80/20 of the case matrix

Recall The invariances, in one breath

Adding a constant to the whole row (Cell C) or scaling it (Cell B) changes $μ$ and $σ$ but not $\hat{x}$ — LayerNorm keeps only the shape of the row. Length-2 rows collapse to $\pm 1$ (Cell E). A flat row (Cell F) — and always a single-feature row (Cell F′, $H = 1$ ) — has $σ = 0$ , so $ϵ$ is mandatory. The affine $γ, β$ (Cell G) can rescale, reshape, or fully undo — no information is lost.

Which two transformations of the input leave

\hat{x}

unchanged?

Scaling every entry by

k

and adding a constant

c

to every entry.

For a length-2 row with

a > b

, what is

\hat{x}

(+ 1, - 1)

— it keeps only the ordering of the two entries and discards their actual values.

Why does a constant row (or any

H = 1

row) need

ϵ > 0

Because

σ^{2} = 0

gives

0/0

(NaN);

ϵ

makes the denominator a safe positive number and the output becomes

0

To fully undo LayerNorm, set

γ

and

β

to what?

γ_{i} = σ^{2} + ϵ

and

β_{i} = μ

, which recovers the original

x_{i}