Exercises — Batch normalization

The mean ${\mu }_{\mathcal{B}}$ comes first. Why? Variance is defined as the average of squared deviations from the mean, ${\sigma }_{\mathcal{B}}^2=\frac{1}{m}{\sum }_i(x_i-{\mu }_{\mathcal{B}}{)}^2$ (recall $m$ = batch size). You literally cannot compute a deviation until you know what you are deviating from. So ${\mu }_{\mathcal{B}}$ is a prerequisite for ${\sigma }_{\mathcal{B}}^2$. In the figure above, the "measure balance point" box must fire before the "measure spread" box.

Training → (b) mini-batch statistics. Each mini-batch $\mathcal{B}$ gets its own ${\mu }_{\mathcal{B}},{\sigma }_{\mathcal{B}}^2$. This makes the output of BN depend on the other examples in the batch — a form of noise that acts as a mild regularizer. Inference → running (population) statistics ${\mu }_{run},{\sigma }_{run}^2$, the exponential moving averages accumulated during training. Why the switch? At test time we may see one example at a time — no batch exists to average over — and we need the output to be deterministic.

False. BN's first act is subtracting the batch mean. Adding a constant $b$ to every activation shifts every $x_i$ and shifts ${\mu }_{\mathcal{B}}$ by exactly the same $b$, so the deviation $x_i-{\mu }_{\mathcal{B}}$ is unchanged. The bias cancels perfectly. Its job is instead done by the learnable $\beta$. Practical rule: set bias=False in the layer feeding into BN.

Step 1 — mean. ${\mu }_{\mathcal{B}}=\frac{2+4+6+8}{4}=\frac{20}{4}=5$. Step 2 — variance (biased, $/m$ with $m=4$). deviations $[-3,-1,1,3]$; squares $[9,1,1,9]$; mean $=\frac{20}{4}=5$. So ${\sigma }_{\mathcal{B}}^2=5$, ${\sigma }_{\mathcal{B}}=\sqrt{5}\approx 2.2360679$. Step 3 — normalize. Since $ϵ=0$, ${\hat{x}}_i=\frac{x_i-5}{\sqrt{5}}$: $\hat{x}=\left[\frac{-3}{\sqrt{5}},\frac{-1}{\sqrt{5}},\frac{1}{\sqrt{5}},\frac{3}{\sqrt{5}}\right]\approx [-1.3416,-0.4472,0.4472,1.3416].$ Sanity: mean of $\hat{x}=0$, variance $=1$. ✓ (Compare to Example 1 in the parent — same shape, just a wider input, so the same standardized pattern comes out.)

Apply $y_i=\gamma {\hat{x}}_i+\beta =3{\hat{x}}_i-1$: $y\approx [3(-1.3416)-1,3(-0.4472)-1,3(0.4472)-1,3(1.3416)-1]=[-5.0249,-2.3416,0.3416,3.0249].$ Why: $\gamma$ stretches the unit-variance signal to standard deviation $3$, and $\beta$ slides the whole thing down by $1$. So the output has mean $\beta =-1$ and standard deviation $|\gamma |=3$.

No batch — use the frozen running stats. $\hat{x}=\frac{18-10}{\sqrt{16}}=\frac{8}{4}=2,y=4\cdot 2+2=10.$ The map $x\mapsto y$ is now a fixed affine function; the same $x=18$ always yields $y=10$ regardless of any neighbours (there are none).

Deviations all $0\Rightarrow {\sigma }_{\mathcal{B}}^2=0$. So ${\hat{x}}_i=\frac{7-7}{\sqrt{0+10^{-5}}}=\frac{0}{0.0031623}=0$ for all $i$. The $ϵ$ turned a lethal $0/0$ into a safe $0$ — exactly the safety role defined at the top of the page. Information destroyed: the common value $7$ — anything constant across the batch is centered to $0$ and erased. Small-batch link: with few examples, batches are more likely to be near-constant or have wildly swinging ${\mu }_{\mathcal{B}},{\sigma }_{\mathcal{B}}^2$ from step to step, so the normalization is noisy/unreliable — motivating Layer Normalization or Group Normalization in that regime.

To undo normalization we invert Step 4: we need ${\gamma }^{\star }{\hat{x}}_i+{\beta }^{\star }=x_i$. Since ${\hat{x}}_i=\frac{x_i-{\mu }_{\mathcal{B}}}{\sqrt{{\sigma }_{\mathcal{B}}^2+ϵ}}$, choosing ${\gamma }^{\star }=\sqrt{{\sigma }_{\mathcal{B}}^2+ϵ}=\sqrt{5}\approx 2.2360679,{\beta }^{\star }={\mu }_{\mathcal{B}}=5$ gives $y_i=\sqrt{5}\cdot \frac{x_i-5}{\sqrt{5}}+5=(x_i-5)+5=x_i$. ✓ Check $x_1=2$: ${\hat{x}}_1=\frac{-3}{\sqrt{5}}\approx -1.3416$; $y_1=2.2360679(-1.3416)+5\approx -3.0+5=2$. ✓ The identity is reachable, so BN adds no restriction — it only changes the optimization landscape.

Why the forms: $y_i=\gamma {\hat{x}}_i+\beta$, so $\frac{\partial y_i}{\partial \gamma }={\hat{x}}_i$ and $\frac{\partial y_i}{\partial \beta }=1$. Both $\gamma$ and $\beta$ are shared across all $m$ examples, so by the chain rule their total gradient is the sum of each example's contribution: $\frac{\partial L}{\partial \gamma }={\sum }_i\frac{\partial L}{\partial y_i}\cdot {\hat{x}}_i$ and $\frac{\partial L}{\partial \beta }={\sum }_i\frac{\partial L}{\partial y_i}\cdot 1$. (See Backpropagation.) Numbers with $\hat{x}\approx [-1.3416,-0.4472,0.4472,1.3416]$: $\frac{\partial L}{\partial \gamma }=0.1(-1.3416)+(-0.2)(-0.4472)+0.3(0.4472)+(-0.4)(1.3416)\approx -\mathrm{0.4472.}$ Term by term: $-0.13416+0.08944+0.13416-0.53664=-0.44720$. So $\frac{\partial L}{\partial \gamma }\approx -0.4472$. $\frac{\partial L}{\partial \beta }=0.1-0.2+0.3-0.4=-\mathrm{0.2.}$

With $b=3$: pre-BN values $=[4,5,6]$. Mean $=5$; deviations $[-1,0,1]$; variance $=\frac{1+0+1}{3}=\frac{2}{3}$; $\sigma =\sqrt{2/3}\approx 0.8165$. Normalized: $[\frac{-1}{0.8165},0,\frac{1}{0.8165}]\approx [-1.2247,0,1.2247]$. With $b=0$: pre-BN values $=[1,2,3]$. Mean $=2$; deviations $[-1,0,1]$; same variance $\frac{2}{3}$; normalized $\approx [-1.2247,0,1.2247]$. Identical. Why: adding $b$ shifted every value and the mean by $+3$, leaving deviations untouched, and variance is a function of deviations only. Conclusion: the bias before BN is dead weight — confirms Exercise 1.3 quantitatively.

Each step: ${\mu }_{run}\leftarrow 0.9{\mu }_{run}+0.1{\mu }_{\mathcal{B}}$.

• After batch 1: $0.9(0)+0.1(10)=1.0$.
• After batch 2: $0.9(1.0)+0.1(12)=0.9+1.2=2.1$.
• After batch 3: $0.9(2.1)+0.1(8)=1.89+0.8=2.69$. Why so far below the true ~10? With small $\alpha$ the running average has long memory and warms up slowly; early in training the estimate lags. This is why running stats are unreliable if used too early, and why some frameworks debias them.

Expand $y=\gamma \frac{x-{\mu }_{run}}{\sqrt{{\sigma }_{run}^2+ϵ}}+\beta$: $y=\underset{a}{\underbrace{\frac{\gamma }{\sqrt{{\sigma }_{run}^2+ϵ}}}}x+\underset{c}{\underbrace{\left(\beta -\frac{\gamma {\mu }_{run}}{\sqrt{{\sigma }_{run}^2+ϵ}}\right)}}.$ $a=\frac{4}{\sqrt{16}}=\frac{4}{4}=1$. $c=2-\frac{4\cdot 10}{4}=2-10=-8$. So $y=x-8$. Check with $x=18$: $18-8=10$ — matches Exercise 2.3. ✓ Because it is pure affine, this $a,c$ can be absorbed into the preceding linear layer's weights and bias, giving BN zero inference cost.

Choose Layer Normalization (Transformers' standard), or Group Normalization for vision with tiny batches. Why not BN: with batch size $2$, ${\mu }_{\mathcal{B}},{\sigma }_{\mathcal{B}}^2$ are estimated from just two numbers — catastrophically noisy (Exercise 3.1's lesson taken to the limit); and single-example inference forces reliance on possibly-unwarmed running stats (Exercise 4.2). The axis rule: BN averages across the batch (same feature, many examples); Layer Norm averages across the features of a single example — so it is completely batch-independent, identical at train and test. That independence is exactly what small-batch / streaming settings need. This connects to Vanishing and Exploding Gradients and Optimization and Conditioning of the Loss Surface: all three normalizers aim at a better-conditioned surface, but by averaging over different axes.

For a constant batch every $x_i=c$, so ${\mu }_{\mathcal{B}}=c$ and each deviation $x_i-{\mu }_{\mathcal{B}}=0$ exactly (not just in the limit). Then ${\hat{x}}_i=\frac{0}{\sqrt{{\sigma }_{\mathcal{B}}^2+ϵ}}=0,y_i=\gamma \cdot 0+\beta =\beta .$ The $\sqrt{\cdot +ϵ}$ denominator is strictly positive (since $ϵ>0$), so no division blows up and the numerator being $0$ forces $y_i=\beta$ regardless of $c$. Interpretation: the value of a constant feature is invisible to BN — it collapses to the learned shift $\beta$. The signal that survives BN is only the relative spread within the batch, which is why BN cannot help when a batch carries no within-batch variation.

Prediction: $b$ is irrelevant (Ex 1.3 / 4.1) — BN removes it. So compute as if pre-BN values were $[0,5,10]$'s the deviations only... but let's include $b=100$ honestly and watch it cancel. Pre-BN: $[100,105,110]$. Mean $=105$. Deviations $[-5,0,5]$. Variance $=\frac{25+0+25}{3}=\frac{50}{3}\approx 16.6667$; $\sigma =\sqrt{50/3}\approx 4.0825$. $\hat{x}=[\frac{-5}{4.0825},0,\frac{5}{4.0825}]\approx [-1.2247,0,1.2247]$. $y=2\hat{x}+1\approx [-1.4495,1.0,3.4495]$. Same $\hat{x}$ (and hence $y$) as if $b=0$ — the huge bias $100$ shifted the mean by $100$ and vanished in the deviations. Prediction confirmed. ✓

Which comes first, mean or variance? ::: Mean — variance needs deviations from it. Training uses which stats? ::: Per-mini-batch ${\mu }_{\mathcal{B}},{\sigma }_{\mathcal{B}}^2$. Inference uses which stats? ::: Running (population) ${\mu }_{run},{\sigma }_{run}^2$. Why drop the bias before BN? ::: BN subtracts the mean, cancelling any constant bias; $\beta$ replaces it. Which estimator for variance? ::: Biased, divide by $m$ (batch size), not $m-1$. Tiny-batch / streaming fix? ::: Use Layer Normalization or Group Normalization.