Residual connections and layer norm placement: Worked examples

This page lives under Residual connections and layer norm placement. We already built the ideas there. Here we compute — we push real numbers through Post-LN and Pre-LN blocks so you can see what each equation does. If you have never divided by a standard deviation before, don't worry: every symbol gets earned here again, from zero.

What the symbols mean (fast rebuild from zero)

Before any example, let us re-anchor every piece of notation to a picture, so no line surprises you.

The whole game of this note: where do we put the LayerNorm box relative to that fork? The two placements are drawn side-by-side below — refer back to it as you work each example.

Post-LN: fork → add → then normalize. $x_{i + 1} = LN (x_{i} + F (x_{i}))$
Pre-LN: normalize inside the branch → add. $x_{i + 1} = x_{i} + F (LN (x_{i}))$

Figure — Residual connections and layer norm placement

What to see in figure s01. Two vertical block diagrams share the picture. On the left (Post-LN, magenta title), information enters at x_in (bottom), splits at the fork: one copy runs through the violet Sublayer F box, and the other — the orange highway — runs to the right and up. They meet at the magenta add box ( $x + F (x)$ ). Crucially the orange highway then feeds into the navy LayerNorm box before reaching x_out at the top — trace it: the highway is swallowed by the wall. On the right (Pre-LN, violet title), the order flips: x_in first enters the navy LayerNorm box, then the violet Sublayer F, and only the branch is normalized — the orange highway shoots straight up the right edge, past every box, untouched, into the magenta add box. Fix these two orange paths in your memory: the left one hits a wall, the right one never does. That single visual difference explains everything in Ex 5.

The scenario matrix

Every worked example below is tagged with the cell it covers. Together they touch every cell.

#	Case class	What could go wrong / be special	Example
A	Ordinary Post-LN forward	normalize a normal residual sum	Ex 1
B	Ordinary Pre-LN forward	LN inside branch, highway untouched	Ex 2
C	Zero / degenerate input ( $σ = 0$ )	all features equal → divide by zero	Ex 3
D	Large-scale sublayer output	high-variance $F (x)$ at init	Ex 4
E	Gradient through the highway	the " $+ 1$ " Jacobian term, both variants	Ex 5
F	Depth accumulation / limiting behaviour	residual stream growing over $N$ layers	Ex 6
G	Real-world word problem	choosing an architecture for a task	Ex 7
H	Exam-style twist	$γ, β, ϵ$ non-trivial, sign traps	Ex 8

Worked example Ex 1 — Cell A: an ordinary Post-LN forward pass

Statement. A single feature vector enters a Post-LN block. Let $x = (1, 3, 5, 7)$ and suppose the sublayer returns $F (x) = (1, - 1, 1, - 1)$ . With $γ = 1, β = 0, ϵ = 0$ , compute $x_{out} = LN (x + F (x))$ .

Forecast: guess — does normalizing wipe out the " $+ F$ " contribution, or keep it? Write down your guess before reading.

Add the residual: $z = x + F (x) = (2, 2, 6, 6)$ . Why this step? Post-LN adds first — the highway and branch merge before any normalizing.
Mean: $μ = \frac{2 + 2 + 6 + 6}{4} = 4$ . Why? We must centre the bars before measuring spread.
Variance: deviations $= (- 2, - 2, 2, 2)$ , squared $= (4, 4, 4, 4)$ , so $σ^{2} = \frac{16}{4} = 4$ , hence $σ = 2$ . Why? $σ$ is the number we divide by to force spread $= 1$ .
Normalize: $z - μ = (- 2, - 2, 2, 2)$ , divide by $σ = 2$ : $x_{out} = (- 1, - 1, 1, 1)$ . Why? This is the LN definition with $γ = 1, β = 0$ .

The bar charts in figure s02 show this exactly: the input sum $z$ (left, uneven bars) becoming the tidy $\pm 1$ output (right).

What to see in figure s02. Left panel: four violet bars for $z = (2, 2, 6, 6)$ , with a dashed orange line at height $4$ marked "mean=4" cutting through them — two bars sit below it, two above. Right panel: the same four features are now magenta bars at exactly $(- 1, - 1, 1, 1)$ , symmetric about a dashed orange line at $0$ marked "mean=0". Compare the two panels: the staircase pattern (small, small, big, big) is identical, but the absolute heights collapsed onto $\pm 1$ . That collapse is LayerNorm erasing scale.

Verify: the output has mean $\frac{- 1 - 1 + 1 + 1}{4} = 0$ ✓ and variance $\frac{1 + 1 + 1 + 1}{4} = 1$ ✓ — exactly what LN promises. Notice the pattern of $F$ survived (the last two entries are larger), but its absolute scale was erased. That erasure is precisely why Post-LN gradients become scale-sensitive.

Worked example Ex 2 — Cell B: the same numbers through Pre-LN

Statement. Same $x = (1, 3, 5, 7)$ . In Pre-LN we first normalize, then the sublayer runs on the normalized vector, then we add the original $x$ back. Suppose the sublayer is the identity for simplicity, $F (u) = u$ . Compute $x_{out} = x + F (LN (x))$ .

Forecast: will the output still have mean $0$ and variance $1$ ? (In Ex 1 it did.)

Normalize the input: $μ = \frac{1 + 3 + 5 + 7}{4} = 4$ , deviations $(- 3, - 1, 1, 3)$ , squared $(9, 1, 1, 9)$ , $σ^{2} = \frac{20}{4} = 5$ , $σ = 5 \approx 2.2360679...$ Why this step? Pre-LN feeds the sublayer clean, centred, unit-spread input.
LN output: $\tilde{x} = \frac{( - 3 , - 1 , 1 , 3 )}{5} \approx (- 1.3416, - 0.4472, 0.4472, 1.3416)$ . Why? This is the completed division from step 1 — the actual normalized vector that the sublayer will receive. We compute it explicitly so we can add it back in step 4.
Sublayer (identity): $F (\tilde{x}) = \tilde{x}$ . Why? We chose identity to isolate the placement effect, not the sublayer's own maths.
Add the untouched highway: $x_{out} = x + \tilde{x} \approx (- 0.3416, 2.5528, 5.4472, 8.3416)$ . Why? The residual stream $x$ is never normalized — this is the whole Pre-LN idea.

Figure s03 stacks the three bar charts: the raw $x$ , the normalized $\tilde{x}$ (centred on zero), and their sum $x_{out}$ (which keeps $x$ 's tall staircase shape).

What to see in figure s03. Three side-by-side panels. Left (violet): the raw $x = (1, 3, 5, 7)$ — a rising staircase all above zero. Middle (orange): $LN (x)$ — the same staircase shape but now centred, straddling zero symmetrically at $(- 1.34, - 0.45, 0.45, 1.34)$ . Right (magenta): the sum $x + LN (x)$ , with a dashed navy line at height $4$ marked "mean=4". Notice the right panel looks like the left panel nudged — the mean stayed at $4$ because the orange middle vector averages to zero and so adds nothing to the average. This is the visual proof that the Pre-LN highway preserves the stream's centre.

Verify: output mean $= \frac{- 0.3416 + 2.5528 + 5.4472 + 8.3416}{4} = 4.0$ ✓ (it kept $x$ 's mean of $4$ , because $\tilde{x}$ has mean $0$ and adds nothing to the average). Contrast with Ex 1 where the mean was forced to $0$ . This is the key difference: Post-LN pins statistics; Pre-LN lets the stream grow.

Worked example Ex 3 — Cell C: degenerate input, all features equal (

$σ = 0$ ) Statement. What does $LN$ do to $x = (5, 5, 5, 5)$ ? Do it once with $ϵ = 0$ and once with $ϵ = 1 0^{- 5}$ .

Forecast: all bars are the same height — is there any "spread" to normalize? What happens to the division?

Mean: $μ = 5$ . Deviations: $(0, 0, 0, 0)$ . Variance: $σ^{2} = 0$ . Why highlight this? This is the degenerate case the parent glossed over. It is real: it happens on a padding/uniform token.
With $ϵ = 0$ : we compute $\frac{0}{0} = \frac{0}{0}$ — undefined. The network would emit NaN. Why this matters? It shows the $ϵ$ cushion is not decoration; it prevents a crash.
With $ϵ = 1 0^{- 5}$ : numerator is $(0, 0, 0, 0)$ , denominator $0 + 1 0^{- 5} \approx 0.00316$ . Result $= \frac{0}{0.00316} = (0, 0, 0, 0)$ . Why? Zero numerator beats a small denominator: a flat vector normalizes to the zero vector, cleanly.

Verify: output $(0, 0, 0, 0)$ has mean $0$ and (defined) variance $0$ ✓. Lesson for the matrix: always keep $ϵ > 0$ ; a constant vector is a legal input and only $ϵ$ makes it survive.

Worked example Ex 4 — Cell D: a large-scale sublayer output at initialization

Statement. At init, random weights make $F (x)$ big. To compare Post-LN vs Pre-LN scale we use $ϵ = 1 0^{- 5}$ throughout this example (so the degenerate LN in the Pre-LN branch is well-defined — remember Ex 3). Take $x = (0, 0, 0, 0)$ (a freshly-initialized zeroed stream) and $F (x) = (100, - 100, 100, - 100)$ .

Note on the sublayer $F$ used here. Modern Transformer sublayers are affine-then-nonlinear maps $F (u) = W_{2} ϕ (W_{1} u + b_{1}) + b_{2}$ . A very common initialization choice sets the biases $b_{1}, b_{2}$ to zero (and for Pre-LN, the last projection's weights are often initialized near zero too — see 6.2.05-Layer-scaling-and-initialization). Under that standard zero-bias init, feeding the all-zeros vector gives $F (0) = W_{2} ϕ (0) + 0$ ; with $ϕ$ a zero-preserving nonlinearity (ReLU/GELU give $ϕ (0) = 0$ ) this yields $F (0) = 0$ . That is the only reason we may write $F (0) \approx 0$ below — it is an initialization fact, not a general property of $F$ .

Forecast: which architecture lets that " $100$ " survive into the next layer, and which one squashes it?

Post-LN: $z = x + F (x) = (100, - 100, 100, - 100)$ . $μ = 0$ , deviations equal $z$ , $σ^{2} = \frac{4 \cdot 10 0 ^{2}}{4} = 10000$ , so $σ^{2} + ϵ = 10000.00001 \approx 100$ . Normalize: $x_{out} \approx (1, - 1, 1, - 1)$ . Why? No matter how huge $F$ is, LN divides it back down to spread $1$ . Good for the forward value — but the backward Jacobian now carries a factor $\frac{γ}{σ ^{2} + ϵ} \approx \frac{1}{100}$ , shrinking gradients by $100 \times$ .
Pre-LN: $\tilde{x} = LN (0) = 0$ (flat input normalizes to zero with the $ϵ$ cushion, exactly as in Ex 3), so the sublayer runs on $\tilde{x} = 0$ . By the zero-bias init above, $F (0) = 0$ , hence $x_{out} = x + F (0) = 0$ . The huge random $100$ never even fires, because in Pre-LN the weights act on normalized (bounded) inputs, not on raw ones. Why? Pre-LN normalizes before the big weights act, so scale explosions are gated at the entrance.

Verify: Post-LN forward magnitude $\approx 1$ (bounded ✓) but gradient scale factor $\approx 1/ 10000 = 1/100$ ✓ — the documented Post-LN gradient shrinkage. Pre-LN keeps the highway at scale $\approx 0$ at init and adds only controlled contributions ✓. This is exactly why Post-LN needs learning-rate warmup and Pre-LN is more forgiving (see 4.2.02-Training-stability-and-convergence).

Worked example Ex 5 — Cell E: the gradient "

$+ 1$ " highway, both variants Statement. For $x_{i + 1} = x_{i} + F (x_{i})$ with a scalar toy $F (x) = 0.3 x$ , find $\frac{d x _{i + 1}}{d x _{i}}$ . Then contrast the Post-LN case where a normalizer scales the whole thing by $1/ σ = 0.5$ .

Reminder on the word "Jacobian". In general a Jacobian is the table of all partial derivatives of a vector output with respect to a vector input. But in this scalar toy model each vector is a single number, so the Jacobian collapses to just the ordinary derivative $\frac{d x _{i + 1}}{d x _{i}}$ — nothing fancier. Read "Jacobian" as "the derivative" throughout this example.

Forecast: which derivative can never fall below $1$ , and which can be dragged toward zero?

Pre-LN derivative: $\frac{d x _{i + 1}}{d x _{i}} = 1 + \frac{d F}{d x _{i}} = 1 + 0.3 = 1.3$ . Why this step? The residual add contributes a guaranteed $1$ ; the branch adds $0.3$ on top. The $1$ is the highway (the straight orange arrow in figure s01) — it cannot be multiplied away.
Chain over $L = 10$ Pre-LN layers (same factor): total $\approx 1. 3^{10} = 13.7858...$ Why? Ten stacked derivatives multiply. Because each is $\geq 1$ , the signal survives depth.
Post-LN with a normalizer factor $0.5$ per layer: each layer's derivative is $0.5 \times 1.3 = 0.65$ ; over $10$ layers $0.6 5^{10} = 0.013462...$ Why? In Post-LN the LN wraps the whole sum (the orange arrow dives into the navy box in s01), so its $\frac{γ}{σ}$ multiplies the $1$ too — the highway loses its protection. $0.65 < 1$ ⇒ vanishing over depth.

Verify: $1. 3^{10} = 13.7858...$ ✓ (grows, gradient survives) and $0.6 5^{10} = 0.01346...$ ✓ (shrinks $\sim 74 \times$ ). This numeric gap is the mechanistic reason Pre-LN scales to very deep nets — see 5.1.03-Gradient-flow-in-deep-networks.

Worked example Ex 6 — Cell F: residual-stream accumulation, limiting behaviour over depth

Statement. In Pre-LN the stream is never normalized, so its variance can grow. Suppose each layer adds a branch output with variance $v = 1$ and the additions are independent. After $N$ layers, what is the stream's variance? Evaluate for $N = 1, 24, 96$ .

Forecast: does the stream variance stay flat, grow linearly, or blow up exponentially?

One layer: $x_{1} = x_{0} + h_{0}$ , and variances of independent sums add: $Var (x_{1}) = Var (x_{0}) + v$ . Start from $Var (x_{0}) = 0$ : after layer $1$ , variance $= 1$ . Why? Independence ⇒ variances add, cross terms average out.
General $N$ : $Var (x_{N}) = N \cdot v = N$ . So $σ_{stream} = N$ . Why? Linear (not exponential) growth — the residual highway accumulates, it does not amplify multiplicatively.
Evaluate: $N = 1 \Rightarrow σ = 1$ ; $N = 24 \Rightarrow σ = 24 \approx 4.8990$ ; $N = 96 \Rightarrow σ = 96 \approx 9.7980$ . Why? Shows the limiting behaviour: the stream grows like $N$ , gently. This is why deep Pre-LN models often add a final LayerNorm before the output head (see 6.2.05-Layer-scaling-and-initialization).

Verify: $24 = 4.89897...$ ✓ and $96 = 9.79795...$ ✓. Growth is $\propto N$ , bounded and predictable — not the exponential blow-up a naive reader might forecast.

Worked example Ex 7 — Cell G: a real-world word problem (choosing the architecture)

Statement. You must train a 96-layer language model from scratch on a single training run with a tight compute budget — you cannot afford many failed restarts, and you want to be forgiving about the learning-rate schedule. Post-LN or Pre-LN? Justify with the numbers from earlier examples.

Forecast: which one gives you the safest single-shot training?

Recall the gradient scaling (Ex 5): Post-LN can shrink gradients by $\sim 74 \times$ over just $10$ layers; over $96$ layers a per-layer factor $0.65$ gives $0.6 5^{96} \approx 1.9 \times 1 0^{- 18}$ — effectively zero gradient in early layers. Why this step? A single failed deep run wastes the whole budget; we must avoid vanishing gradients.
Recall Pre-LN (Ex 5): a per-layer factor $\geq 1$ keeps early-layer gradients alive across all $96$ layers. Why? The protected " $+ 1$ " highway is depth-uniform.
Decision: choose Pre-LN. It is more forgiving of the learning rate and warmup (though warmup is still used at this scale — GPT-3/PaLM keep it). Why? Matches the parent note's corrected claim: Pre-LN reduces warmup dependence, it does not eliminate warmup.

Verify: $0.6 5^{96} \approx 1.9 \times 1 0^{- 18}$ (checked ✓) confirming Post-LN's deep-layer gradient death, so Pre-LN is the sound single-shot choice. Cross-reference 4.2.02-Training-stability-and-convergence.

Worked example Ex 8 — Cell H: exam twist with non-trivial

$γ, β, ϵ$ and a sign trap Statement. Compute $LN (x)$ for $x = (- 2, 0, 2, 4)$ with $γ = (2, 2, 2, 2)$ , $β = (1, 1, 1, 1)$ , $ϵ = 0$ . Trap: a student forgets $γ, β$ and forgets to keep the sign of deviations.

Forecast: will the negative entry stay negative after scale-and-shift? Guess the first output entry.

Mean: $μ = \frac{- 2 + 0 + 2 + 4}{4} = \frac{4}{4} = 1$ . Why? Centre first — including the negative value, which the trap-student mishandles.
Deviations (keep signs!): $(- 2 - 1, 0 - 1, 2 - 1, 4 - 1) = (- 3, - 1, 1, 3)$ . Variance: squares $(9, 1, 1, 9)$ , $σ^{2} = \frac{20}{4} = 5$ , $σ = 5 \approx 2.23607$ . Why? Signs matter in the numerator even though variance squares them away.
Normalize: $\frac{( - 3 , - 1 , 1 , 3 )}{5} \approx (- 1.3416, - 0.4472, 0.4472, 1.3416)$ . Why? This is the raw normalized vector, before the knobs act.
Scale by $γ = 2$ , then shift by $β = 1$ (multiply each entry by $2$ , then add $1$ ):
- entry 1: $2 (- 1.3416) + 1 = - 1.6833$
- entry 2: $2 (- 0.4472) + 1 = 0.1056$
- entry 3: $2 (0.4472) + 1 = 1.8944$
- entry 4: $2 (1.3416) + 1 = 3.6833$
so $x_{out} \approx (- 1.6833, 0.1056, 1.8944, 3.6833)$ . Why? $γ, β$ are the learnable knobs applied after normalization — the trap-student who dropped them gets the wrong scale, and one who flips signs gets entry 1 positive by mistake.

Verify: first entry $2 (- 1.3416) + 1 = - 1.6833$ ✓ (still negative — the sign survived, refuting the "everything becomes positive" trap). Before $γ, β$ the vector has mean $0$ , variance $1$ ; after the knobs, mean $= β = 1$ ✓ and variance $= γ^{2} = 4$ ✓ — the knobs re-scale the distribution exactly as designed.

Recall Self-check (reveal after guessing)

Post-LN forces output mean/variance to ::: $0$ and $1$ (statistics are pinned every layer). Pre-LN leaves the residual stream ::: un-normalized, so it accumulates (variance grows like $N$ ). The role of $ϵ$ in $σ^{2} + ϵ$ is ::: to prevent division by zero when all features are equal ( $σ = 0$ ). The gradient term that Pre-LN protects but Post-LN does not is ::: the " $+ 1$ " identity Jacobian of the residual add. Over 10 layers, per-layer derivative $1.3$ vs $0.65$ gives ::: $\approx 13.79$ (survives) vs $\approx 0.0135$ (vanishes).

Related deep dives & prerequisites: 4.1.01-Self-attention-mechanism, 4.1.08-Multi-head-attention, 4.3.01-Positional-encoding, 4.1.12-Encoder-decoder-attention, 5.1.03-Gradient-flow-in-deep-networks, 6.2.05-Layer-scaling-and-initialization, 4.2.02-Training-stability-and-convergence.