Residual connections and layer norm placement: Question bank

This is the misconception-hunting page for Residual connections and layer norm placement. Each line is a trap the topic sets. Cover the answer, commit to a reason, then reveal. The goal is not to memorize "Pre-LN good" — it is to be able to say why at the level of gradients, variance, and the residual stream.

Prerequisites worth re-reading if a line stings: 5.1.03-Gradient-flow-in-deep-networks, 4.2.02-Training-stability-and-convergence, 6.2.05-Layer-scaling-and-initialization.

True or false — justify

Residual connections make the network deeper.

False. They change how gradients and information flow through a fixed depth; the layer count is unchanged. They let deep networks train, they don't add depth.

Without a residual connection the gradient always vanishes.

False — it can vanish OR explode. Each backward matrix multiply can shrink or amplify the gradient; residuals guard against both by injecting an identity (

+ 1

) path that neither shrinks nor grows.

In Pre-LN, the main residual stream

x_{i}

is itself normalized at every layer.

False. LayerNorm sits inside the branch on

LayerNorm (x_{i})

; the stream

x_{i}

is never normalized, which is exactly why it accumulates information additively across layers.

Post-LN normalizes the sublayer output before adding the residual.

False. Post-LN adds first (

z_{i} = x_{i} + Sublayer (x_{i})

) then normalizes:

x_{i + 1} = LayerNorm (z_{i})

. The normalization sits on the sum, which is why it bottlenecks the main gradient.

Layer norm and batch norm are interchangeable in Transformers.

False. BatchNorm's statistics depend on other examples in the batch and on padding/sequence length; LayerNorm uses only the single example's features, so it is stable across variable-length sequences.

The learnable

γ

and

β

in LayerNorm are optional cosmetic parameters.

False. Without them LayerNorm forces every output to zero mean and unit variance, destroying representational scale;

γ, β

let the layer re-learn any needed scale and shift.

Pre-LN completely eliminates the need for learning-rate warmup.

False. Pre-LN makes training more forgiving of warmup choice; large models (GPT-3, PaLM) still use warmup. The correct claim is "less dependent on warmup," not "no warmup."

The

+ 1

\frac{d y}{d x} = 1 + \frac{d F}{d x}

holds even before the sublayer

F

has learned anything.

True. That is the whole point: even if

F (x) \approx 0

at init, the identity gradient still flows, so early training doesn't stall.

Post-LN gradients are uniform across all layers at initialization.

False. Xiong et al. (2020) show Post-LN gradients are large and uneven across depth at init — precisely the reason warmup is needed to survive early steps.

LayerNorm normalizes across the batch dimension.

False. It normalizes across the feature dimension (

d

features) per example. The mean

μ

and variance

σ^{2}

are computed over one token's feature vector.

Spot the error

"Residuals help because

y = x \cdot F (x)

keeps a multiplicative path open."

The error is multiplicative. Residuals are additive:

y = x + F (x)

. Addition gives the clean

+ 1

Jacobian term; multiplication would reintroduce the very shrink/explode problem residuals exist to avoid.

"In Post-LN,

x_{i + 1} = LayerNorm (x_{i}) + Sublayer (x_{i})

Wrong grouping. LayerNorm wraps the whole sum:

x_{i + 1} = LayerNorm (x_{i} + Sublayer (x_{i}))

. Normalizing only

x_{i}

would be neither Pre-LN nor Post-LN.

"Pre-LN's Jacobian is

\frac{\partial x _{i + 1}}{\partial x _{i}} = \frac{\partial h _{i}}{\partial x _{i}}

Missing the identity. Because

x_{i + 1} = x_{i} + h_{i}

, the Jacobian is

I + \frac{\partial h _{i}}{\partial x _{i}}

. Dropping the

I

erases the entire gradient-highway argument.

"We add

ϵ

inside the square root just to match the paper's notation."

No —

ϵ

prevents division by zero when

σ^{2} \to 0

(a constant feature vector). It is numerical safety, not decoration.

"LayerNorm's Jacobian has unbounded entries, so it can't bottleneck gradients."

Backwards on both counts. Its entries are bounded by the division by

σ

, and that bounding is the bottleneck: it dampens large updates to

z_{i}

in Post-LN.

"Since Pre-LN normalizes the input, the sublayer output

h_{i}

is guaranteed zero-mean, so the stream stays bounded."

The input to the sublayer is normalized, not the output. The output

h_{i}

has whatever scale the sublayer produces, and the un-normalized stream

x_{i}

grows as these add up — that growth is a known Pre-LN property, not a contradiction.

"Because Pre-LN is more stable, it always reaches lower final loss than Post-LN."

Not guaranteed. Post-LN, when it trains, can match or beat Pre-LN quality; Pre-LN's advantage is trainability/stability at depth, not a universal quality win.

Why questions

Why does the

+ 1

term specifically prevent vanishing gradients?

Because the backward pass multiplies Jacobians layer by layer; a factor of exactly

1

can be chained through any number of layers without decay, so at least an identity-strength gradient always reaches early layers.

Why is Post-LN training "delicate" at initialization?

Random-scale sublayer outputs give the sum

z_{i}

high, uneven variance; LayerNorm-on-the-sum makes backward gradients scale-sensitive and non-uniform across depth, so large early steps can diverge.

Why does putting LayerNorm inside the branch fix the gradient highway?

The normalization then sits on

LayerNorm (x_{i})

inside

h_{i}

, not on the sum

x_{i} + h_{i}

. The main path keeps its untouched

I

Jacobian, so gradients flow through unchanged while the branch's contribution merely adds in.

Why LayerNorm and not BatchNorm for sequence models?

Sequences vary in length and use padding; batch statistics would leak that structure into every token. LayerNorm depends only on one example's features, staying identical whether the batch has one sequence or a thousand.

Why does the "residual stream" interpretation require Pre-LN, not Post-LN?

A residual stream means each layer reads from and writes to a shared, un-normalized accumulator. Post-LN re-normalizes after every add, resetting the stream's scale each layer, so there is no persistent additive stream to read from.

Why does warmup specifically help Post-LN?

Warmup ramps the learning rate up slowly, so the model can settle into a well-behaved activation regime before the large, uneven Post-LN gradients get multiplied by a big step size and blow up.

Why is per-example normalization important when sequences are batched together with padding?

Padded positions carry no real information; if normalization mixed them in (as batch stats would), real tokens' normalized values would depend on how much padding happened to sit beside them.

Edge cases

What does LayerNorm output if every feature in

x

is identical (a constant vector)?

Then

σ^{2} = 0

, so

\frac{x - μ}{0 + ϵ} = 0

for every entry; the output is just the shift

β

. The

ϵ

is what keeps this finite instead of

0/0

What happens to gradient flow if the sublayer

F

outputs exactly zero everywhere?

In Pre-LN,

x_{i + 1} = x_{i}

and the Jacobian is

I

— the layer is a perfect identity and the gradient passes through untouched. This is the safe "do nothing" default residuals provide.

As depth grows very large, which architecture degrades and why?

Post-LN degrades: its gradient imbalance across depth compounds, making stable training progressively harder without careful init and tuning. Pre-LN's depth-uniform gradients scale far better.

In Pre-LN, does the un-normalized residual stream growing in magnitude with depth break the model?

No — it's expected. Each sublayer normalizes its input via LayerNorm regardless of the stream's current scale, so a growing stream is read correctly; a final LayerNorm typically cleans up the output.

If you set

γ = 1, β = 0

and freeze them, is LayerNorm now a pure identity?

No — it still subtracts the mean and divides by

σ

. It becomes standardization without learnable rescale, not an identity map.

At the very first sublayer of the network, what does the "

x_{i}

" in the residual add refer to?

The embedded (and positionally encoded, see 4.3.01-Positional-encoding) input tokens — the residual stream is seeded by the input embeddings, and every layer thereafter adds to it.

Does the encoder–decoder cross-attention sublayer also get a residual + LayerNorm wrapper?

Yes. Every sublayer — self-attention, cross-attention, and FFN — is wrapped identically; the residual/LayerNorm pattern is per-sublayer, not per-layer.

Recall Self-test before you close

One-line challenges — answer out loud, then check yourself.

The exact Post-LN formula for one sublayer is? ::: $x_{i + 1} = LayerNorm (x_{i} + Sublayer (x_{i}))$ . The exact Pre-LN formula for one sublayer is? ::: $x_{i + 1} = x_{i} + Sublayer (LayerNorm (x_{i}))$ . The Pre-LN residual-path Jacobian is? ::: $I + \frac{\partial h _{i}}{\partial x _{i}}$ — the $I$ is the highway. Over which dimension does LayerNorm compute $μ, σ^{2}$ ? ::: The feature dimension $d$ , per single example. One reason Pre-LN still keeps warmup? ::: Large-scale training remains more stable with it; Pre-LN only loosens the dependence, it doesn't remove it.