Residual connections and layer norm placement

4.1.9 · AI-ML › Transformer Architecture

Overview

Residual connections (skip connections) aur layer normalization ka placement Transformers mein critical architectural choices hain jo training stability, gradient flow, aur final performance determine karte hain. In dono mechanisms ke beech interaction non-obvious hai: LayerNorm ko residual path ke relative kahan rakhte ho, yeh fundamentally badal deta hai ki gradients kaise propagate hote hain aur model kaise seekhta hai.

Figure — Residual connections and layer norm placement

Core Concepts

Do Architectures: Post-LN vs Pre-LN

Post-LN (Original Transformer, 2017)

Formula Post-LN Architecture Derivation

Yeh structure kyun? Original "Attention is All You Need" paper ne LayerNorm ko residual addition ke baad place kiya. Chaliye derive karte hain ki gradient flow ke liye iska kya matlab hai.

Ek single sublayer ke liye: $x_{i + 1} = LayerNorm (x_{i} + Sublayer (x_{i}))$

Step 1: Sublayer transformation compute karo $h_{i} = Sublayer (x_{i}) (attention or FFN)$

Step 2: Residual connection add karo $z_{i} = x_{i} + h_{i}$

Step 3: LayerNorm apply karo $x_{i + 1} = LayerNorm (z_{i}) = γ ⊙ \frac{z _{i} - μ ( z _{i} )}{σ ^{2} ( z _{i} ) + ϵ} + β$

Gradient flow analysis: Backprop ke dauran, gradient ko LayerNorm ke normalization se pass karna padta hai. LayerNorm ka Jacobian bounded entries rakhta hai ( $σ$ se division ki wajah se), lekin yeh ek bottleneck create karta hai: $z_{i}$ ke large updates normalization se dampen ho jaate hain.

Critical issue: Initialization par, agar $Sublayer (x_{i})$ large values produce karta hai, toh residual sum $z_{i}$ high variance rakhta hai. LayerNorm ise normalize karega, lekin gradients activations ke scale ke sensitive ho jaate hain. Yahi ek reason hai ki Post-LN training delicate hoti hai aur typically careful initialization aur learning-rate warmup par rely karti hai.

Post-LN structure ek full layer ke liye (attention + FFN):

y z = LayerNorm (x + MultiHeadAttention (x)) = LayerNorm (y + FFN (y))

Worked example Post-LN Training Dynamics

Scenario: Ek deep Post-LN Transformer (jaise 12+ layers) scratch se train karna.

Jo generally observe hota hai (Xiong et al., 2020, "On Layer Normalization in the Transformer Architecture" ke analysis se):

Early training fragile hoti hai: Warmup ke bina, Post-LN training diverge kar sakti hai, kyunki initialization par deeper layers mein expected gradient magnitude bada aur depth ke across uneven hota hai.
Warmup important hai: Post-LN Transformers typically ek learning-rate warmup schedule ke saath train hote hain (LR ko kuch thousands of steps mein small value se ramp up karna) precisely is early instability se bachne ke liye.
Depth ke across gradient imbalance: Theoretical aur empirical analyses dikhate hain ki Post-LN gradients initialization par layers ke across uniform nahi hote, yahi warmup ko necessary banata hai.

Yeh step kyun? Initialization par sublayer outputs essentially random scale rakhte hain. LayerNorm-after-residual backward gradients ko scale-sensitive aur depth ke across uneven banata hai. Warmup model ko stable activation regime mein settle hone deta hai bade steps lene se pehle.

Pre-LN (Modern Standard)

Formula Pre-LN Architecture Derivation

Yeh structure kyun? Pre-LN LayerNorm ko pehle sublayer se, residual branch ke andar apply karta hai. Yeh design (Xiong et al., 2020 dwara formally analyze kiya gaya) large models ke liye standard ban gaya kyunki yeh initialization par gradient behavior improve karta hai.

Ek single sublayer ke liye: $x_{i + 1} = x_{i} + Sublayer (LayerNorm (x_{i}))$

Step 1: Input par LayerNorm apply karo $\tilde{x}_{i} = LayerNorm (x_{i})$

Yeh step kyun? Sublayer ke input ko normalize karo taaki usse controlled, zero-mean, unit-variance activations milein chahe previous layers mein kuch bhi hua ho.

Step 2: Normalized input par sublayer transformation compute karo $h_{i} = Sublayer (\tilde{x}_{i})$

Step 3: Residual connection add karo (yahan LayerNorm nahi!) $x_{i + 1} = x_{i} + h_{i}$

Gradient flow analysis: Backprop ke dauran gradient residual path se ek clean "+1" term ke saath flow karta hai Jacobian mein: $\frac{\partial x _{i + 1}}{\partial x _{i}} = I + \frac{\partial h _{i}}{\partial x _{i}}$

LayerNorm branch ke andar hai, toh yeh main gradient highway ko bottleneck nahi karta. Yeh bahut zyada stable, aur depth-uniform gradient flow create karta hai—gradient largely unchanged residual path se flow kar sakta hai jabki normalized branch gradients usmein add hote hain.

Key advantage: Residual stream $x_{i}$ kabhi normalize nahi hota, toh yeh information additively saari layers ke across accumulate karta hai. Yeh ek "residual stream" interpretation create karta hai: har layer ek shared information stream se padhta hai aur usmein likhta hai.

Pre-LN structure ek full layer ke liye:

y z = x + MultiHeadAttention (LayerNorm (x)) = y + FFN (LayerNorm (y))

Worked example Pre-LN Training Dynamics

Scenario: Usi depth ka ek deep Pre-LN Transformer. Jo generally observe hota hai:

Zyada stable early training: Loss training ke shuru se smoothly decrease hoti hai, aur model learning rate ke choice ke baare mein kaafi zyada robust hota hai.
Warmup relax ho sakta hai lekin ELIMINATE nahi hota: Pre-LN warmup par dependence kam karta hai, lekin bade modern Pre-LN models (GPT-3, PaLM, etc.) stable large-scale training ke liye abhi bhi learning-rate warmup schedules use karte hain. Sahi statement hai "Pre-LN warmup choice ke baare mein zyada forgiving hai," na ki "Pre-LN ko koi warmup nahi chahiye."
Depth ke across zyada uniform gradient norms: Xiong et al. dikhate hain ki expected gradient magnitude layers ke across bahut better behaved hoti hai, jo mechanistic reason hai ki Pre-LN bahut deep networks tak scale karta hai.

Yeh kyun kaam karta hai: LayerNorm har sublayer ke input ko normalize karta hai, toh har layer ko well-behaved activations milti hain. Residual path ek clean gradient highway provide karta hai. Model initialization aur warmup details ke baare mein bahut kam sensitive hota hai.

Practical difference: Bahut deep, bahut large models (GPT-3, PaLM) train karne mein practice mein Pre-LN use hota hai. Post-LN progressively mushkil hota jaata hai stable train karne ke liye jaise depth barhti hai bina careful tuning ke.

Mathematical Comparison: Gradient Flow

Formula Gradient Flow Derivation

Chaliye exactly derive karte hain ki dono architectures mein gradients kaise flow karte hain. Maano hamare paas loss $L$ hai aur hum $\frac{\partial L}{\partial x _{i}}$ chahte hain given $\frac{\partial L}{\partial x _{i + 1}}$ .

Post-LN Gradient:

$x_{i + 1} = LN (x_{i} + F (x_{i}))$ Chain rule se: $\frac{\partial L}{\partial x _{i}} = \frac{\partial L}{\partial x _{i + 1}} \frac{\partial x _{i + 1}}{\partial ( x _{i} + F ( x _{i} ))} (I + \frac{\partial F}{\partial x _{i}})$

LayerNorm ka Jacobian $\frac{\partial LN}{\partial z}$ , $z = x_{i} + F (x_{i})$ ke statistics par depend karta hai: $\frac{\partial LN ( z )}{\partial z _{j}} = \frac{γ}{σ} (δ_{ij} - \frac{1}{d} - \frac{( z _{i} - μ ) ( z _{j} - μ )}{σ ^{2}})$

Key observation: Is Jacobian mein $O (1/ d)$ off-diagonal terms hain aur yeh $z$ ke variance $σ^{2}$ par depend karta hai. Agar $F (x_{i})$ large outputs produce karta hai, $σ$ bada hoga, aur gradient $1/ σ$ se scale down ho jaayega.

Pre-LN Gradient:

$x_{i + 1} = x_{i} + F (LN (x_{i}))$

Chain rule se: $\frac{\partial L}{\partial x _{i}} = \frac{\partial L}{\partial x _{i + 1}} (I + \frac{\partial F}{\partial LN ( x _{i} )} \frac{\partial LN ( x _{i} )}{\partial x _{i}})$

Critical difference: Gradient $\frac{\partial L}{\partial x _{i + 1}}$ directly identity term $I$ se flow karta hai bina kisi normalization ke. LayerNorm Jacobian sirf branch gradient ko affect karta hai, main pathway ko nahi.

Yeh kyun matter karta hai: Ek 24-layer Post-LN Transformer mein main gradient ko har layer par ek LayerNorm Jacobian se pass karna padta hai, jiska scaling har layer par activation variance $σ$ par depend karta hai. Pre-LN mein, main gradient path ek simple additive accumulation hai—highway par koi per-layer multiplicative normalization nahi.

Qualitative estimate: Xiong et al. (2020) dikhate hain ki Pre-LN mein initialization par expected gradient norm essentially depth se independent hai, jabki Post-LN mein yeh depth ke saath ek aisa tarika barhta hai jis mein deeper layers dominate karti hain—yahi depth-dependence exactly hai jise warmup compensate karta hai. (Koi specific decay constant jaise $0. 8^{L}$ quote karne se bachein; precise factor architecture aur initialization par depend karta hai aur ek universal number nahi hai.)

Practical Implications

Worked example Real-World Architecture Choices

Original Transformer / early BERT / GPT-2: Post-LN use kiya. Yeh relatively shallow hain (12–24 layers) aur learning-rate warmup aur gradient clipping ke saath train hote hain.

GPT-3 (2020) aur baad ke large models: Pre-LN use karte hain, jo bahut deep networks aur bahut large parameter counts tak scale karne par kaafi zyada stable hai. Yeh models abhi bhi warmup schedules use karte hain—Pre-LN warmup ke sensitivity ko kam karta hai, ise remove nahi karta.

T5 (2019): Standard LayerNorm ke saath Pre-LN-style placement use karta hai (T5 kuch implementations mein ek LayerNorm variant use karta hai jo bias/mean-subtraction step omit karta hai, lekin widely stated fact yeh hai ki yeh LayerNorm use karta hai, RMSNorm nahi).

PaLM (2022): FFN mein SwiGLU activations ke saath Pre-LN. Pre-LN structure training instability mein hit hue bina new activation functions ke saath experiment karna aasaan banata hai, aur ise ek warmup schedule ke saath train kiya jaata hai.

Common mistake Layer Norm Placement ke Baare mein Common Misconceptions

Mistake 1: "Post-LN output ko normalize karta hai, jo training ko zyada stable banana chahiye."

Kyun sahi lagta hai: Residual add ke baad normalization lagta hai jaise yeh activation scales ko better control karega—tum har layer ke final output ko normalize kar rahe ho.

Fix: Yeh intuition ulta hai! Output ko normalize karne ka matlab hai ki backprop ke dauran gradients ko normalization se guzarna padta hai, main pathway par. Yeh ek bottleneck aur depth-dependent gradient imbalance create karta hai. Pre-LN har sublayer ke input ko normalize karta hai (jahan yeh actually help karta hai) jabki main gradient path ko clean rakhta hai.

Steel-man: Post-LN forward activation scales ko tightly control karta hai. Lekin deep learning mein, gradient flow forward activation control se zyada matter karta hai. Pre-LN kuch forward activation control sacrifice karta hai (residual stream unnormalized hai) bahut better gradient flow ke liye.

Mistake 2: "Pre-LN aur Post-LN same solution par converge karne chahiye kyunki woh same operations ko sirf rearrange kar rahe hain."

Kyun sahi lagta hai: Dono architectures mein same LayerNorm aur residual connections hain, bas alag order mein. Surely woh same function class express karte hain?

Fix: Order optimization ke liye profoundly matter karta hai, na sirf expressiveness ke liye. Post-LN initialization par depth-dependent gradients ke saath ek difficult optimization landscape create karta hai. Pre-LN ek smoother landscape create karta hai. Woh similar functions express kar sakte hain, lekin ek optimize karna kaafi aasaan hai—especially large depth par.

Mistake 3: "Pre-LN ko bilkul bhi learning-rate warmup ki zaroorat nahi hai."

Kyun sahi lagta hai: Pre-LN clearly training ko zyada stable aur warmup-sensitive kam banata hai, toh yeh conclude karna tempting hai ki warmup unnecessary hai.

Fix: Pre-LN warmup par dependence reduce karta hai aur kabhi kabhi small scale par iske bina train ho sakta hai, lekin large models (GPT-3, PaLM, etc.) practice mein abhi bhi warmup schedules use karte hain. Accurate claim hai "Pre-LN warmup choice ke baare mein zyada robust hai," na ki "warmup unnecessary hai."

Mistake 4: "Residual connection $x$ add karta hai, toh output scale linearly depth ke saath barhti hai."

Kyun sahi lagta hai: Pre-LN mein, hum $x$ mein add karte rehte hain: $L$ layers ke baad, kya scale $O (L)$ times badi nahi ho jaani chahiye?

Fix: Variance roughly linearly barhti hai, scale nahi. Agar har sublayer roughly variance $σ^{2}$ ke saath ek independent contribution add karta hai, toh $L$ layers ke baad total variance $O (L σ^{2})$ hogi, toh standard deviation $O (L)$ se barhti hai. Yeh manageable hai—aur sublayers ko small output variance ke saath initialize karke control kiya ja sakta hai (final projection ko scale down karna).

Yeh actually ek feature hai: Residual stream ki variance growth gradual hai ( $L$ ). Post-LN mein, LayerNorm variance ko roughly 1 rakhta hai har layer par, jo acha lagta hai lekin model ko depth ke across alag information ke liye alag activation scales use karne se rokta hai.

Advanced: The Residual Stream Interpretation

Intuition Residual Stream as Information Highway

Pre-LN Transformers mein, hum residual path $x$ ko ek stream ki tarah soch sakte hain jo saari layers ke across information carry karta hai:

Layer 1 stream se padhti hai, attention compute karti hai, stream mein wapas likhti hai.
Layer 2 updated stream padhti hai, updated info par attention compute karti hai, wapas likhti hai.
...
Layer 24 fully processed stream padhti hai, predictions output karti hai.

Har layer ek "read-modify-write" operation perform karti hai. Stream kabhi reset ya normalize nahi hoti—yeh ek persistent memory hai. Yeh interpretation explain karti hai ki Pre-LN kyun kaam karta hai: gradient is stream ke through backward flow kar sakta hai bina kisi bottleneck ke.

Analogy: Socho ek relay race jahan runners ek baton pass karte hain (residual stream). Pre-LN mein, har runner apna contribution add karta hai aur aage pass karta hai—baton information accumulate karta hai. Post-LN mein, har runner baton ko ek "reset station" (LayerNorm) se pass karta hai jo ise standardize karta hai—yeh information accumulation prevent karta hai aur gradient issues create karta hai.

Implementation Details

Worked example Pre-LN Transformer Block (PyTorch-style pseudocode)

class PreLNTransformerBlock:
   def forward(self, x):
       # Self-attention with Pre-LN
       residual = x
       x = layer_norm_1(x)              # Normalize before attention
       x = multi_head_attention(x)      # Attention on normalized input
       x = residual + x                 # Add to unnormalized residual
       # Feedforward with Pre-LN
       residual = x
       x = layer_norm_2(x)              # Normalize before FFN
       x = feedforward(x)               # FFN on normalized input
       x = residual + x                 # Add to unnormalized residual
       return x

Yeh step kyun? Pattern hai: x = x + sublayer(layernorm(x)). Residual hamesha unnormalized $x$ mein add hota hai, aur LayerNorm ek copy par apply hota hai jo sublayer mein jaata hai.

Initialization detail: Ek common trick hai ki har sublayer mein final linear projection (attention output projection aur FFN second layer) ko small scale ke saath initialize karo (jaise $L$ layers ke liye $1/ 2 L$ ke proportional) taaki residual stream variance depth ke across roughly controlled rahe.

Connections

4.1.01-Self-attention-mechanism: Residual connections self-attention blocks ke around wrap hote hain
4.1.08-Multi-head-attention: Har head ka output project aur residual stream mein add hota hai
4.2.02-Training-stability-and-convergence: Pre-LN deep models mein training stability dramatically improve karta hai
4.3.01-Positional-encoding: Residual stream positional information layers ke across carry karta hai
5.1.03-Gradient-flow-in-deep-networks: Core gradient flow concepts jo explain karte hain Pre-LN kyun kaam karta hai
6.2.05-Layer-scaling-and-initialization: Residual connections ke liye proper initialization
4.1.12-Encoder-decoder-attention: Encoder-decoder cross-attention ke liye same Pre-LN pattern use karta hai

Active Recall

Recall Ek 12-saal ke bachche ko explain karo: Skip connections kyun chahiye?

Socho tum 24 logon ke saath telephone ka game khel rahe ho. Normally, message end tak pahunchte pahunchte garbled ho jaata hai—har insaan thoda sunne mein galti kar sakta hai aur use thoda badal sakta hai.

Ab socho agar pehla insaan directly aakhri insaan ko whisper kar sake, aur beech ke sabhi log bhi aakhri insaan ko whisper kar sakein. Aakhri insaan original message clearly aur saare changes jo sabne kiye sunta hai. Woh ek skip connection hai!

Neural networks mein, skip connections original information ko seedha baad ki layers mein travel karne deti hain bina beech ke saare math se "garbled" hue. Yeh network ko better seekhne mein help karta hai kyunki woh original input ko processed version se compare kar sakta hai aur seekh sakta hai ki kaun se changes actually helpful the.

#flashcards/ai-ml

Transformers mein residual connection ka mathematical form kya hai?

output = input + Sublayer (input)

, jahan input directly transformed output mein add hota hai, ek identity gradient path create karta hai.

Pre-LN Post-LN se better gradient flow kyun provide karta hai?

Pre-LN mein, gradients residual path se ek clean identity term (

I + ...

) ke saath main pathway par flow karte hain, jabki Post-LN mein gradients ko har layer par LayerNorm ke Jacobian se pass karna padta hai, depth-dependent, scale-sensitive bottlenecks create karte hain.

Ek Transformer layer ke liye Pre-LN architecture formula likho

y = x + Attention (LN (x))

phir

z = y + FFN (LN (y))

, jahan LayerNorm pehle residual branch ke andar har sublayer ke apply hota hai.

Ek Transformer layer ke liye Post-LN architecture formula likho

y = LN (x + Attention (x))

phir

z = LN (y + FFN (y))

, jahan LayerNorm residual addition ke baad apply hota hai.

Residual gradient

\frac{d y}{d x} = 1 + \frac{d F}{d x}

mein "+1" term kya guarantee karta hai?

Yeh guarantee karta hai ki kam se kam ek identity gradient backward flow kare, vanishing gradients prevent karta hai chahe sublayer

F

kuch na seekhe ya initially near-zero gradients ho.

Kya Pre-LN learning-rate warmup ki zaroorat eliminate kar deta hai?

Nahi. Pre-LN warmup ke sensitivity ko reduce karta hai, lekin GPT-3 aur