Visual walkthrough — Residual connections and layer norm placement

---

Step 1 — What a layer even is: a box that changes a list of numbers

WHAT. A Transformer processes each token as a vector — an ordered list of $d$ numbers (for real models $d$ is 512, 768, ...). Call that list $x$. A "sublayer" is just a box $F$ that eats $x$ and produces another list of the same length, $F(x)$. The box is either multi-head attention or a feed-forward network — but for this whole page we only care that it is some function.

WHY. Before we can talk about "adding the input back" or "where LayerNorm goes", we need one clean mental object: a list of numbers going into a box and a list coming out. Everything else is arranging boxes.

PICTURE. Look at the figure: the blue arrow is the input list $x$ flowing into the orange box $F$. Out comes $F(x)$, a new list. Nothing has been added yet — this is a plain transformation.

---

Step 2 — The residual: don't replace the input, add to it

WHAT. Instead of throwing $x$ away and keeping only $F(x)$, we do $y=\underset{\text{the original list, untouched}}{\underbrace{x}}+\underset{\text{what the box computed}}{\underbrace{F(x)}}$ We add the two lists number-by-number. The output $y$ is "the input, plus a correction $F$ wanted to make".

WHY. Two reasons, and both are drawable. (1) If the box learns nothing useful at the start, $F(x)\approx 0$ and $y\approx x$ — the layer harmlessly passes information through instead of destroying it. (2) It creates a straight path from $x$ to $y$ that skips the box entirely. Hold that straight path in mind; it is the whole point of Step 4.

PICTURE. The green arrow is the new character: it carries $x$ around the box and drops it onto the sum at the circle marked $+$. The box's output (orange) also flows into that circle. The straight green line is the skip connection.

---

Step 3 — Why "add" gives a free +1 backward (building the derivative from scratch)

WHAT. Training asks a backward question: if I nudge the input $x$ by a tiny amount, how much does the output $y$ move? That ratio "output-wiggle per input-wiggle" is the derivative, written $\frac{\partial y}{\partial x}$. For lists it is a grid of such ratios called the Jacobian $J$ — entry $J_{ij}$ says how much output-number $i$ moves when input-number $j$ wiggles.

Differentiate $y=x+F(x)$ term by term: $\frac{\partial y}{\partial x}=\underset{\text{from the }x\text{ term}}{\underbrace{I}}+\underset{\text{from the box}}{\underbrace{\frac{\partial F}{\partial x}}}.$ Here $I$ is the identity matrix — the Jacobian of "copy $x$ unchanged" is a grid with $1$s on the diagonal and $0$s off it, because nudging input-number $j$ moves output-number $j$ by exactly the same amount and touches no other.

WHY the derivative and not something simpler? Because "how much the error at the output blames the input" is a rate of change, and rate of change is precisely what a derivative measures. No other tool answers "wiggle in → wiggle out". We need $I$ separated from $\partial F/\partial x$ because that lone $I$ is the guarantee.

PICTURE. The green skip path contributes the flat $I$ (a diagonal of $1$s, drawn as a highlighted diagonal). The orange box path contributes $\partial F/\partial x$, which can be tiny or huge. During backprop the gradient multiplies by this sum, so even if the box's part collapses to zero, the $+I$ keeps a full-strength signal flowing. That is the highway.

---

Step 4 — LayerNorm: the box that flattens a list to zero-mean, unit-spread

WHAT. LayerNorm takes a list $z$ and re-centres and re-scales it so its numbers have average $0$ and spread $1$, then lets the model re-stretch and re-shift with learned knobs $\gamma ,\beta$: $\text{LN}(z)=\gamma \odot \underset{\text{flatten to mean }0,\text{spread }1}{\underbrace{\frac{z-\mu }{\sqrt{{\sigma }^2+ϵ}}}}+\beta .$

• $\mu =\frac{1}{d}{\sum }_i{z}_i$ — the mean, the balance point of the list.
• ${\sigma }^2=\frac{1}{d}{\sum }_i(z_i-\mu {)}^2$ — the variance, average squared distance from the balance point (how spread out the numbers are).
• $\sqrt{{\sigma }^2+ϵ}$ — the standard deviation, a plain "typical distance"; $ϵ$ is a tiny safety number so we never divide by zero.
• $\odot$ means multiply lists element-by-element; $\gamma ,\beta$ are learned per-feature dials.

WHY subtract-then-divide? Subtracting $\mu$ slides the whole list so its centre sits at $0$; dividing by $\sigma$ squeezes or stretches it to a standard width of $1$. A neural box behaves best when its inputs are neither drifting off to huge numbers nor squashed near zero — LayerNorm forces that good regime per token, independent of sequence length (why not Batch Norm: batch stats depend on padding; see training stability).

PICTURE. Left: a wild input list, numbers scattered far from centre. After subtracting $\mu$ the pile slides onto $0$; after dividing by $\sigma$ it is squeezed to width $1$. The output (green) is tidy.

---

Step 5 — Post-LN: normalize after the add (the LayerNorm sits ON the highway)

WHAT. The original 2017 Transformer wraps the whole residual sum inside LayerNorm: $x_{i+1}=\text{LN}\left(\underset{\text{residual sum }{z}_i}{\underbrace{x_i+F(x_i)}}\right).$ So the sequence is: box → add the skip → then normalise everything.

WHY look at this first? Because it is the natural first guess ("normalise the layer's output") and it exposes the danger. Differentiate it: $\frac{\partial x_{i+1}}{\partial x_i}=\underset{\text{LayerNorm’s Jacobian}}{\underbrace{J_{\text{LN}}(z_i)}}(I+\frac{\partial F}{\partial x_i}).$ The clean $I+\partial F/\partial x$ from Step 3 is now multiplied on the left by $J_{\text{LN}}$. That means our precious $+I$ never travels alone — it is first passed through the LayerNorm Jacobian, whose entries shrink with the spread $\sigma$ of $z_i$. At initialization the box output is random-scaled, $\sigma$ varies, so the effective highway strength varies across depth.

PICTURE. The LayerNorm box (blue) straddles the merge point — every backward arrow, including the green skip's arrow, must pass through it. The highway has a tollbooth on it. Notice the deeper layers get an uneven gradient size (drawn as thicker/thinner arrows).

---

Step 6 — Pre-LN: normalize before the box (the highway stays clean)

WHAT. Move LayerNorm inside the branch, before the box, and add the raw $x_i$ back: $x_{i+1}=\underset{\text{raw, un-normalized}}{\underbrace{x_i}}+F(\text{LN}(x_i)).$ Sequence: normalise a copy → box → add the original $x_i$ (not the normalised one) back.

WHY this fixes it. Differentiate: $\frac{\partial x_{i+1}}{\partial x_i}=\underset{\text{from the raw }{x}_i}{\underbrace{I}}+\underset{\text{branch, LN buried inside}}{\underbrace{\frac{\partial F}{\partial x_i}{J}_{\text{LN}}(x_i)}}.$ The $I$ now stands alone and un-multiplied — nothing sits on the green highway. LayerNorm's Jacobian is trapped inside the branch term, so it only affects the correction the box adds, never the guaranteed identity flow. The gradient reaches every depth at full strength; norms stay uniform across layers.

PICTURE. The LayerNorm box has moved off the highway and onto the orange branch only. The green skip line runs from $x_i$ straight to the sum, tollbooth-free. Backward arrows on the green line are all equally thick at every depth.

---

Step 7 — Edge & degenerate cases (never leave the reader stranded)

WHAT / WHY / PICTURE for each corner:

• The box learns nothing, $F\equiv 0$. Then $\partial F/\partial x=0$. Post-LN gives $J_{\text{LN}}\cdot I=J_{\text{LN}}$ (still scaled/​bottlenecked). Pre-LN gives $I+0=I$ — a perfect identity pass. Pre-LN degrades gracefully to "do nothing", Post-LN does not.
• Spread collapses, $\sigma \to 0$ (all features equal). Division would blow up; the $+ϵ$ under the root keeps $\sqrt{{\sigma }^2+ϵ}\to \sqrt{ϵ}$ finite. Both architectures survive only because of $ϵ$ — this is what that tiny constant is for.
• Huge box output at init (variance of $F$ large). Post-LN: $z_i$ has wild $\sigma$, $J_{\text{LN}}$ swings, deep-layer gradients uneven → divergence without warmup. Pre-LN: raw $x_i$ path is untouched, only the branch is affected → stays stable. See layer scaling & init.
• Very deep stack ($N$ layers). Post-LN multiplies $N$ Jacobians $J_{\text{LN}}$ along the highway → product can shrink/​grow exponentially. Pre-LN's highway is a sum of $I$'s → the raw signal survives to the top no matter how deep.

PICTURE. Two mini-stacks side by side: Post-LN's backward signal (red) fades with depth; Pre-LN's (green) stays constant. The tollbooth-per-layer versus clean-lane contrast is the message.

---

The one-picture summary

Everything above compressed: two blocks side by side. Post-LN — box, add skip, then LayerNorm on the merged highway; backward gradient = $J_{\text{LN}}(I+\partial F)$, the $I$ is trapped behind a tollbooth. Pre-LN — LayerNorm on the branch only, raw skip straight through; backward gradient = $I+\partial F{J}_{\text{LN}}$, a lone clean $I$. The green highway is unbroken only on the right.

Recall Feynman retelling — say it back in plain words

Every layer is a box that rewrites a list of numbers. Instead of trusting the box, we keep the original list and add the box's change to it — that's the residual. When we ask "how much does the output move if the input wiggles", the answer splits into two pieces: a guaranteed "$1$" from the kept-original copy, plus whatever the box contributes. That guaranteed $1$ is a highway the training signal can always drive down, so deep networks stop losing their gradient.

LayerNorm is a separate box that tidies a list to average-zero, spread-one so the next box gets clean input. The only question is where you park it. Park it after the add (Post-LN) and it sits on the highway — now the guaranteed $1$ has to squeeze through LayerNorm first, its strength wobbles with the data, deep layers get uneven gradients, and you must warm the learning rate up slowly or training explodes. Park it before the box (Pre-LN) and it only touches the branch, never the highway; the guaranteed $1$ rides through untouched at every depth. Same two ingredients, different parking spot, completely different training life.

Corner cases: a tiny $ϵ$ stops division-by-zero if the list is flat; if the box does nothing, Pre-LN passes the input perfectly while Post-LN still scales it; stack it very deep and Post-LN multiplies many tollbooths together (bad) while Pre-LN just keeps adding clean $1$'s (good).

Recall

In $y=x+F(x)$, what is the Jacobian and why does it help? ::: $I+\partial F/\partial x$; the constant $I$ never vanishes, so a full-strength gradient always flows backward. Where does Post-LN place LayerNorm relative to the add? ::: After the residual add — LayerNorm wraps $x_i+F(x_i)$, sitting on the highway. Where does Pre-LN place LayerNorm? ::: Before the sublayer, inside the branch; the raw $x_i$ is added back un-normalized. Why does Pre-LN give more uniform gradients across depth? ::: Its backward Jacobian is $I+\partial F{J}_{LN}$ — the $I$ stands alone, so the residual highway is never bottlenecked by LayerNorm. What is the tiny $ϵ$ inside LayerNorm for? ::: To keep $\sqrt{{\sigma }^2+ϵ}$ finite when the variance ${\sigma }^2\to 0$, preventing division by zero. Does Pre-LN eliminate learning-rate warmup? ::: No — it is more forgiving of warmup, but large models still use warmup.

---

Prerequisites & neighbours: 4.1.01-Self-attention-mechanism · 4.1.08-Multi-head-attention · 4.3.01-Positional-encoding · 4.1.12-Encoder-decoder-attention · 5.1.03-Gradient-flow-in-deep-networks · 6.2.05-Layer-scaling-and-initialization · 4.2.02-Training-stability-and-convergence