Layer normalization

3.2.9 · AI-ML › Training Deep Networks

YE EXIST KYU KARTA HAI?

Deep networks kai layers stack karte hain. Layer $\ell$ apne weights sikhti hai yeh assume karke ki inputs ki ek certain distribution hogi. Lekin woh inputs layer $\ell -1$ ke outputs hain, jinke weights bhi change ho rahe hain. Isliye layer $\ell$ ka input distribution training ke dauran shift hota rehta hai — informally ise internal covariate shift kehte hain. Bade ya lopsided activations se exploding/vanishing gradients bhi hote hain aur learning slow ho jaati hai.

BatchNorm ne yeh batch dimension across normalize karke fix kiya. Lekin isse har example ki statistics uske batch-mates se tie ho jaati hain. Yeh tab break hota hai jab:

• batch tiny ho (noisy statistics),
• aap ek time pe ek token process kar rahe ho (RNNs / autoregressive Transformers),
• train vs. inference statistics alag hon.

KAISE: scratch se derive karo

Humare paas ek example ka pre-normalization vector $\mathbf{x}\in {\mathbb{R}}^H$ hai.

Step 1 — features pe mean. Kyun? Zero pe recenter karne ke liye taaki layer high ya low biased na ho. $\mu =\frac{1}{H}{\sum }_{i=1}^H{x}_i$

Step 2 — features pe variance. Kyun? Spread measure karne ke liye taaki hum ek controlled size pe rescale kar sakein. ${\sigma }^2=\frac{1}{H}{\sum }_{i=1}^H(x_i-\mu {)}^2$

Step 3 — standardize karo. Kyun? $\mu$ subtract karna ise center karta hai; $\sqrt{{\sigma }^2+ϵ}$ se divide karna variance ko $\approx 1$ karta hai. $ϵ$ (jaise $10^{-5}$) division by zero se bachata hai jab ek vector near-constant ho. ${\hat{x}}_i=\frac{x_i-\mu }{\sqrt{{\sigma }^2+ϵ}}$

Step 4 — affine restore. Kyun? Mean 0 / var 1 force karna bahut rigid ho sakta hai (jaise ek sigmoid ko wider range mein inputs chahiye). Isliye hum har feature ko ek learnable scale ${\gamma }_i$ aur shift ${\beta }_i$ dete hain. Agar best cheez yeh ho ki normalize na karen, toh network ${\gamma }_i=\sqrt{{\sigma }^2+ϵ}$, ${\beta }_i=\mu$ seekh sakta hai aur original recover kar sakta hai. $y_i={\gamma }_i{\hat{x}}_i+{\beta }_i$

BatchNorm vs LayerNorm — crucial axis

Worked examples

Common mistakes (steel-manned)

The 80/20

• LayerNorm = ek example ke features across normalize karo → mean 0, var 1 → phir learnable $\gamma ,\beta$.
• Yeh batch-independent hai ⇒ size-1 batches, RNNs, Transformers ke liye kaam karta hai, aur test time pe identical hai.
• $ϵ$ divide-by-zero prevent karta hai; $\gamma ,\beta$ expressiveness restore karte hain.

Flashcards

Connections

• Batch normalization — batch-axis sibling; normalization dimension contrast karo.
• Internal covariate shift — woh problem jise normalization mitigate karta hai.
• Transformers — LayerNorm attention & feed-forward sublayers ke around apply hota hai.
• Residual connections — LayerNorm ke saath Add & Norm ke roop mein combine hota hai.
• RMSNorm — ek sasta variant jo mean-centering drop karta hai.
• Vanishing and exploding gradients — kyun stable activation scales help karte hain.
• Activation functions — kyun $\gamma ,\beta$ next nonlinearity ke liye matter karte hain.

Concept Map