Residual connections and layer norm placement: Foundations

This page assumes you have seen none of the notation used in the parent note. We build every symbol from the ground up, in an order where each piece leans only on pieces already built.

1. What is a "vector of activations"? The symbol $x$

Everything in a Transformer travels as a list of numbers. When we write $x$ , we mean one such list attached to one word (one token).

Figure — Residual connections and layer norm placement

Why the topic needs it. Residual connections add two of these lists together, and layer norm rescales one of these lists. Until you picture $x$ as a bar-chart of $d$ dials, none of the later operations mean anything.

The subscript $x_{j}$ ::: picks out the value on dial number $j$ inside one vector
$d$ ::: how many features (dials) the vector has — the "width" of the model

2. A "sublayer" and the symbol $F (x)$ or $Sublayer (x)$

Inside each floor of the tower sits a machine that takes $x$ in and produces a new list of the same length. In the parent note this machine is written $F (x)$ or $Sublayer (x)$ .

Notation warning. The parent uses three names for the same idea:

symbol	meaning
$F (x)$	generic sublayer in the math derivations
$Sublayer (x)$	the same thing spelled out in words
$MultiHeadAttention (x)$ , $FFN (x)$	the two concrete choices for $F$

Why can we write $x + F (x)$ at all ::: because $F$ returns a vector of the same length $d$ as $x$

3. The plus sign that changes everything: the residual $x + F (x)$

Why the topic needs it — the ONE reason. Recall from the opening callout that a learning arrow (the gradient) has to travel backward through the tower. The around road hands that arrow back untouched. Look at the figure: the arrow can always take the skip road home even if the $F$ road is broken. This is the "gradient highway" the whole topic is about (deepened in 5.1.03-Gradient-flow-in-deep-networks).

The word "skip" refers to ::: the copy of $x$ that goes around $F$ instead of through it

4. Mean, variance, and the symbols $μ$ , $σ^{2}$ , $σ$

Before we can define layer norm we need three plain-arithmetic ideas about a single vector $x$ .

First, one piece of shorthand we are about to use. The symbol $\sum_{j = 1}^{d} x_{j}$ just means "add up, letting $j$ run from $1$ to $d$ ." It is nothing more than $x_{1} + x_{2} + \dots + x_{d}$ written compactly. Keep that in mind for the next two definitions.

Why we subtract $μ$ inside the variance ::: to measure spread relative to the center, not relative to zero
What $σ$ tells you in one phrase ::: the typical size of a feature's deviation from the mean

5. Putting it together: LayerNorm and $γ$ , $β$ , $ϵ$ , $⊙$

Let us earn every new symbol here:

symbol	plain words	picture
$x - μ$	slide all bars down until they balance at zero	bars re-centered on the axis
$σ^{2} + ϵ$	the spread (with a tiny safety number)	the width we divide by
$\frac{x - μ}{\dots}$	now bars have unit spread	a "standardized" bar-chart
$ϵ$	tiny constant, e.g. $1 0^{- 5}$	guards against dividing by $0$ when $σ = 0$
$γ$	learnable scale, one per feature	how tall each bar is allowed to grow
$β$	learnable shift, one per feature	where each bar's baseline sits
$⊙$	multiply element-by-element	dial $j$ times $γ_{j}$ , no mixing

What $γ$ and $β$ let the model do after standardizing ::: re-stretch ( $γ$ ) and re-shift ( $β$ ) each feature, so normalization does not force mean 0 / std 1 if the model prefers otherwise
What breaks if $ϵ = 0$ and all features are equal ::: division by zero, because $σ = 0$

6. The two placements — the whole point of the parent note

Now that $x + F (x)$ and $LayerNorm$ both mean something, the only remaining choice is the order. The parent note calls these Post-LN and Pre-LN.

Here we finally meet the floor subscript $i$ . As warned in §1, this $i$ is not a feature number — it is the number of the floor in the tower. So $x_{i}$ is the whole vector arriving at floor $i$ , and $x_{i + 1}$ is the whole vector leaving for the next floor up. (Inside any one of these vectors the individual dials are still $x_{i, 1}, x_{i, 2}, \dots$ , but we won't need that detail.)

In Pre-LN, is the residual stream $x_{i}$ ever normalized ::: no — LayerNorm sits only inside the branch, so the highway is untouched
The subscript $i$ in $x_{i}$ means ::: which layer/floor of the tower we are on (a whole vector) — not a single feature

How these foundations feed the topic

Read this as a bottom-up build order. Each item uses only the ones above it, and the last line is the parent topic itself:

Activation vector $x$ (§1) — the list of $d$ dials everything is made of.
Sublayer $F (x)$ (§2) — leans on §1: it eats a vector, returns a same-length vector.
Residual $x + F (x)$ (§3) — leans on §1 and §2: only possible because lengths match; it builds the gradient highway.
Mean $μ$ and variance $σ^{2}$ (§4) — lean on §1: pure arithmetic on the dials.
LayerNorm (§5) — leans on §4: uses $μ, σ$ to standardize, then $γ, β$ to rescale; keeps numbers sane.
Placement: Post-LN vs Pre-LN (§6) — leans on §3 and §5: it is nothing but the choice of where to slot LayerNorm relative to the residual — which is exactly the parent topic.

The gradient consequences of that choice connect onward to 5.1.03-Gradient-flow-in-deep-networks, 6.2.05-Layer-scaling-and-initialization and 4.2.02-Training-stability-and-convergence.

Equipment checklist

I can read $x = (x_{1}, \dots, x_{d})$ as a bar-chart of $d$ dials ::: yes — each $x_{j}$ is one dial's height
I can say in one line what the "learning signal / gradient" is ::: a little arrow per dial saying turn-up-or-down and by how much, passed backward down the tower
I know why $x + F (x)$ is even allowed ::: because $F$ returns a same-length vector, so addition makes sense
I can compute $μ$ and $σ^{2}$ of a small vector by hand ::: mean = average of features; variance = average squared distance from the mean
I can state what $γ$ , $β$ , $ϵ$ , $⊙$ each do in LayerNorm ::: scale, shift, divide-by-zero guard, element-wise multiply
I can tell Post-LN from Pre-LN by looking at where $LayerNorm$ sits ::: Post-LN wraps the whole sum; Pre-LN wraps only the input to $F$ , leaving the skip road clean
I know the difference between $x_{j}$ (a feature) and $x_{i}$ (a whole layer's vector) ::: $x_{j}$ is one dial inside a vector; $x_{i}$ is the entire vector at floor $i$
I know the failure case that forces $ϵ > 0$ ::: a flat vector where every feature is equal, giving $σ = 0$