Layer normalization: Question bank — AI-ML deep dive

The recurring characters below:

$x = (x_{1}, \dots, x_{H})$ — one example's activation row of $H$ features.
$μ, σ^{2}$ — mean and variance computed across those $H$ features (never across the batch).
$\overset{x}{^}_{i} = \frac{x _{i} - μ}{σ ^{2} + ϵ}$ — the standardized value; $ϵ$ is a tiny safety number.
$γ_{i}, β_{i}$ — learnable scale and shift applied after standardizing.

True or false — justify

LayerNorm and BatchNorm both compute a mean, so they normalize over the same numbers.

False. Both compute a mean, but BatchNorm averages one feature down a column (across examples) while LayerNorm averages all features across a row (one example) — same tool, orthogonal axis.

LayerNorm behaves identically at training and inference time.

True. Its statistics

μ, σ^{2}

come only from the current example, so there is nothing batch-dependent to freeze — no running averages like BatchNorm needs.

Scaling an entire input vector by

10

changes the LayerNorm output

\overset{x}{^}

(before

γ, β

False. Multiplying every

x_{i}

c

multiplies both the deviation

x_{i} - μ

and

σ^{2}

c

, so they cancel —

\overset{x}{^}

is invariant to scaling the whole row (with

ϵ \to 0

Adding a constant to every feature of the input changes

\overset{x}{^}

False. Adding

c

shifts

μ

by the same

c

, so

x_{i} - μ

is unchanged — LayerNorm is invariant to shifting the whole row too.

The learnable

γ, β

are redundant once we force mean

0

, variance

1

False. A rigid unit-variance range can be wrong for the next nonlinearity;

γ, β

restore expressiveness and can even undo normalization entirely if that helps.

With

γ_{i} = σ^{2} + ϵ

and

β_{i} = μ

, LayerNorm returns the original input.

True. Then

y_{i} = σ^{2} + ϵ \cdot \frac{x _{i} - μ}{σ ^{2} + ϵ} + μ = x_{i}

— the network can learn to be the identity, so LayerNorm never forces harm.

LayerNorm requires a batch of at least a few dozen examples to give stable statistics.

False. It uses only the current example's

H

features, so batch size 1 works perfectly — that batch-independence is its whole selling point in Transformers.

Setting

ϵ = 0

is fine as long as your data is "reasonable".

False in general. If any row is (near) constant,

σ^{2} \to 0

and you divide by

\approx 0

, producing NaNs and exploded gradients.

ϵ

is a guard, not a tuning knob.

After Step 3 (ε=0), the

H

standardized features have mean exactly

0

and variance exactly

1

True.

\frac{1}{H} \sum \overset{x}{^}_{i} = 0

by construction and

\frac{1}{H} \sum \overset{x}{^}_{i}^{2} = σ^{2} / σ^{2} = 1

— a purely algebraic identity, independent of the data values.

RMSNorm and LayerNorm produce the same output on any input.

False. RMSNorm skips mean-subtraction and divides only by the root-mean-square, so it is not invariant to shifting the row; it matches LayerNorm only when

μ

already equals

0

Spot the error

"LayerNorm fixes internal covariate shift by making every feature's distribution match across the batch."

The batch/feature axes are swapped. LayerNorm stabilizes the per-example distribution across its own features; it never looks at other examples, so it cannot equalize a feature across a batch.

"We normalize to unit variance, therefore the output activations are guaranteed to stay unit-variance going into the next layer."

The learnable

γ

(and the next layer's weights) rescale the signal afterward, so post-

γ, β

activations need not be unit-variance — normalization is a starting point, not a permanent constraint.

"LayerNorm needs a running mean and variance stored for test time, just like BatchNorm."

No stored statistics exist. Each example computes its own

μ, σ^{2}

live, so train and inference are byte-for-byte the same procedure.

"Because

ϵ

1 0^{- 5}

, it's safe to say

\overset{x}{^}_{i} = (x_{i} - μ) / σ

The approximation is fine numerically but conceptually the

ϵ

lives inside the square root:

σ^{2} + ϵ

, not

σ^{2} + ϵ

— the placement matters for the degenerate near-constant case.

"LayerNorm removes the need for residual connections since both stabilize training."

They solve different problems: residuals give gradients a shortcut path (fighting vanishing gradients through depth); LayerNorm controls the scale/center of activations. Modern Transformers use both together.

"Since standardizing gives mean

0

, the

β

shift can only ever make things worse."

β

lets the network re-introduce a nonzero mean when the downstream activation prefers it (e.g. feeding a ReLU); it adds freedom, and the net learns

β = 0

if zero-mean is actually best.

Why questions

Why does LayerNorm normalize across features instead of across the batch?

To make each example self-contained, so statistics don't depend on batch-mates — enabling batch size 1, per-token autoregressive decoding, and identical train/test behaviour.

Why is

ϵ

placed inside the square root rather than added afterward?

So that when

σ^{2} \to 0

the denominator smoothly floors at

ϵ

(a small positive number), keeping the division well-defined instead of blowing up.

Why can LayerNorm afford to ignore the overall scale of the incoming vector?

Because an earlier layer multiplying all its outputs by

10

cancels in

\frac{x _{i} - μ}{σ ^{2}}

, so that scale explosion never reaches the next layer — exactly the stability we wanted.

Why do Transformers prefer LayerNorm over BatchNorm specifically?

Transformers process variable-length sequences token by token, where a "batch" of activations is unreliable; LayerNorm's batch-independence and train=test consistency fit that regime.

Why does the affine step use per-feature

γ_{i}, β_{i}

rather than a single shared scalar?

Different features may need different scales/shifts to suit the next layer, so per-feature parameters give the network fine-grained control to reshape each dimension independently.

Why doesn't forcing variance

1

destroy the information in the vector?

Standardizing is an invertible affine map on the row (subtract, divide) plus the invertible

γ, β

restore — relative differences between features survive; only the overall center and scale are made controllable.

Edge cases

A row is exactly constant,

x = (5, 5, 5, 5)

. What does LayerNorm output (γ=1, β=0)?

μ = 5

, so every deviation is

0

and

σ^{2} = 0

; then

\overset{x}{^}_{i} = 0/ ϵ = 0

. The output is the all-zero vector —

ϵ

prevents a NaN and the constant collapses to zero.

H = 1

(a single feature). What happens?

μ = x_{1}

, so

x_{1} - μ = 0

and

σ^{2} = 0

; the output is

0

(then

β_{1}

). LayerNorm is essentially meaningless for one feature — there is no spread across features to normalize.

Two rows that differ only by an overall shift, e.g.

(1, 2, 3)

and

(101, 102, 103)

. Same

\overset{x}{^}

Yes. Shifting a whole row by a constant cancels in

x_{i} - μ

, so both standardize to the identical

\overset{x}{^}

— LayerNorm is shift-invariant per row.

A row with one huge outlier,

(0, 0, 0, 1000)

. Does LayerNorm tame it?

Partially. The outlier inflates

σ^{2}

, so dividing shrinks all values, but the shape (one dominant feature) survives — LayerNorm rescales, it doesn't clip or remove outliers.

Batch of size 1 during inference. Any problem for LayerNorm?

None. It never used the batch dimension, so a single example is processed exactly as during training — unlike BatchNorm, which would have degenerate batch statistics.

Feeding LayerNorm's output into a saturating activation like tanh with tiny

γ

With small

γ

the inputs stay near

0

where tanh is roughly linear, so gradients flow well; the network learns

γ

to place activations in the useful part of the nonlinearity, which is why

γ

isn't redundant.

Recall One-sentence self-test

If I can't state which axis LayerNorm averages over and why that axis makes it batch-independent, I haven't learned the topic — everything else (ε, γ, β, invariances) hangs off that single fact.