Consider a deep network. Layer ℓ receives inputs from layer ℓ−1. Gradient descent updates all layers simultaneously using gradients computed as if the other layers were frozen. But they're not frozen — so after one step, the inputs feeding layer ℓ have a different mean and spread.
WHY this hurts:
If inputs to a sigmoid/tanh drift to large magnitude, activations saturate → gradients vanish.
Different scales across features make the loss surface ill-conditioned (long narrow valleys), forcing tiny learning rates.
HOW BN helps: by re-centering and re-scaling each activation every step, the layer above always sees a well-behaved distribution, so we can use larger learning rates and worry less about initialization.
We want each activation, per mini-batch, to have mean 0 and variance 1. So compute the statistics of the mini-batch.
Let the mini-batch be B={x1,…,xm} (one scalar activation, m examples).
Step 1 — batch mean.Why? We need the center to subtract.
μB=m1∑i=1mxi
Step 2 — batch variance.Why? We need the spread to divide by.
σB2=m1∑i=1m(xi−μB)2
Step 3 — normalize.Why the ϵ? To avoid dividing by zero when a batch happens to be constant; it keeps things numerically stable.
x^i=σB2+ϵxi−μB
Now x^ has (approximately) mean 0, variance 1.
Step 4 — scale and shift.Why undo what we just did? Forcing mean 0/var 1 may destroy useful representational power (e.g. a sigmoid needs inputs away from 0 to be nonlinear). So we give the network the freedom to learn the best mean/scale via two learnable parameters:
yi=γx^i+β
BN is fully differentiable, so gradients flow through μB and σB2 too. The parameter gradients are simple:
∂γ∂L=∑i∂yi∂Lx^i,∂β∂L=∑i∂yi∂LWhy these forms? Since yi=γx^i+β, differentiate: ∂yi/∂γ=x^i and ∂yi/∂β=1, then sum over the batch (both params are shared across all m examples).
Imagine a relay race where each runner passes a baton, but the height they hold it at keeps changing every round, so the next runner keeps fumbling. Batch Norm is a rule: "before passing, always hold the baton at the same standard height." Now everyone knows what to expect and the race speeds up. And if a runner really needs a different height, we let them adjust it on purpose (that's the γ,β dials) — but from a known starting point, not chaos.
Dekho, deep network mein har layer apne se pehle wali layers ka output leti hai. Training ke dauran jab saari layers ke weights update hote hain, toh kisi bhi layer ke input ki distribution (mean aur spread) badalti rehti hai — isko internal covariate shift kehte hain. Matlab har layer ek hilte hue target ko chase kar rahi hai, jisse training slow aur unstable ho jaati hai, learning rate chhota rakhna padta hai.
Batch Normalization ka jugaad simple hai: har mini-batch ke liye us activation ka mean nikaalo, variance nikaalo, phir normalize kar do taaki mean 0 aur variance 1 ho jaaye. Lekin sirf normalize karne se network ki power ghat sakti hai, isliye do learnable parameters — γ (scale) aur β (shift) — de dete hain, taaki network apni pasand ki range khud seekh le. Chaaho toh network identity bhi wapas la sakta hai, so kuch loss nahi hota.
Ek important baat: training time pe batch ke apne stats use hote hain, but test time pe ek hi example aa sakta hai, toh hum training ke dauran ka running average (mean/variance) use karte hain. Isse test output deterministic ho jaata hai. Aur haan — BN se pehle wali layer ka bias hata do, kyunki BN mean subtract karke usko cancel kar deta hai, β hi uska kaam kar deta hai.
Kyun important hai? Kyunki BN se tum bade learning rate use kar sakte ho, training fast aur stable hoti hai, initialization pe kam depend karti hai, aur thoda regularization bhi free mein milta hai. Bas dhyan rakho — bahut chhote batch size (jaise 2) pe BN ke stats noisy ho jaate hain, tab LayerNorm ya GroupNorm behtar hai.
