Batch normalization

3.2.8 · AI-ML › Training Deep Networks

Batch Normalization KYA hai?

Humein iska zaroorat kyun hai? (Problem ko steel-man karna)

Ek deep network consider karo. Layer $\ell$ ko layer $\ell -1$ se inputs milte hain. Gradient descent saari layers ko simultaneously update karta hai, un gradients se jo aise compute hote hain jaise baaki layers frozen hoon. Lekin woh frozen nahi hain — toh ek step ke baad, layer $\ell$ ko milne wale inputs ka mean aur spread alag ho jaata hai.

Yeh kyun nuksan karta hai:

• Agar sigmoid/tanh ke inputs bade magnitude par drift kar jayein, toh activations saturate ho jaate hain → gradients vanish ho jaate hain.
• Features ke beech alag-alag scales ki wajah se loss surface ill-conditioned ho jaata hai (lambi, sankri valleys), jo bahut chhoti learning rates par majboor karta hai.

BN kaise help karta hai: har step par har activation ko re-center aur re-scale karke, upar waali layer ko hamesha ek well-behaved distribution milti hai, toh hum larger learning rates use kar sakte hain aur initialization ki zyada chinta nahi karni padti.

KAISE: BN ko first principles se derive karna

Hum chahte hain ki har activation, har mini-batch ke liye, mean 0 aur variance 1 rakh sake. Toh mini-batch ke statistics compute karte hain.

Maano mini-batch hai $\mathcal{B}=\{x_1,\dots ,x_m\}$ (ek scalar activation, $m$ examples).

Step 1 — batch mean. Kyun? Subtract karne ke liye center chahiye. ${\mu }_{\mathcal{B}}=\frac{1}{m}{\sum }_{i=1}^m{x}_i$

Step 2 — batch variance. Kyun? Divide karne ke liye spread chahiye. ${\sigma }_{\mathcal{B}}^2=\frac{1}{m}{\sum }_{i=1}^m(x_i-{\mu }_{\mathcal{B}}{)}^2$

Step 3 — normalize. $ϵ$ kyun? Jab ek batch constant ho toh zero se divide karne se bachne ke liye; yeh numerical stability banaye rakhta hai. ${\hat{x}}_i=\frac{x_i-{\mu }_{\mathcal{B}}}{\sqrt{{\sigma }_{\mathcal{B}}^2+ϵ}}$ Ab $\hat{x}$ ka (approximately) mean 0, variance 1 hai.

Step 4 — scale aur shift. Jo humne kiya use kyun undo karein? Mean 0/var 1 force karna useful representational power destroy kar sakta hai (jaise sigmoid ko nonlinear rehne ke liye 0 se door inputs chahiye). Toh hum network ko do learnable parameters ke zariye best mean/scale seekhne ki freedom dete hain: $y_i=\gamma {\hat{x}}_i+\beta$

Train vs. Test — ek subtle lekin crucial point

BN ke through Backpropagation (yeh phir bhi train kyun karta hai)

BN fully differentiable hai, toh gradients ${\mu }_{\mathcal{B}}$ aur ${\sigma }_{\mathcal{B}}^2$ se bhi flow karte hain. Parameter gradients simple hain: $\frac{\partial L}{\partial \gamma }={\sum }_i\frac{\partial L}{\partial y_i}{\hat{x}}_i,\frac{\partial L}{\partial \beta }={\sum }_i\frac{\partial L}{\partial y_i}$ Yeh forms kyun? Kyunki $y_i=\gamma {\hat{x}}_i+\beta$ hai, differentiate karo: $\partial y_i/\partial \gamma ={\hat{x}}_i$ aur $\partial y_i/\partial \beta =1$, phir batch par sum karo (dono params saare $m$ examples mein shared hain).

Worked Examples

Common Mistakes (Steel-manned)

Benefits summary (the 80/20)

Flashcards

Connections

• Training Deep Networks
• Layer Normalization
• Group Normalization
• Vanishing and Exploding Gradients
• Weight Initialization
• Dropout
• Optimization and Conditioning of the Loss Surface
• Backpropagation

Concept Map