Visual walkthrough — Batch normalization

We follow one tiny mini-batch of activations through all four operations, then handle the two cases that break naive BN (a constant batch, and single-example inference).

---

Step 1 — The problem, drawn as a cloud of dots

WHAT. We have a mini-batch: a small handful of numbers that one neuron produced, one per training example. Call them $x_1,x_2,\dots ,x_m$. Here $m$ is just how many examples are in the batch. We will use $m=5$ with values $x=[2,4,4,6,9]$.

WHY. A neuron's job is to feed the next layer. But as training proceeds, these numbers drift — their center and their spread wander around. "Center" = where the dots sit on the number line on average. "Spread" = how far apart they are. A wandering center and spread is exactly the internal covariate shift the parent note warned about. To fix a drifting thing, we must first measure it.

PICTURE. The dots below live on a horizontal number line. Notice two facts we will attack in the next two steps: (1) the cloud's middle is nowhere near $0$, and (2) the cloud is wide.

---

Step 2 — Find the center (the mean ${\mu }_{\mathcal{B}}$)

WHAT. Add all the numbers up, divide by how many there are. That balancing point is the mean.

${\mu }_{\mathcal{B}}=\underset{\text{"divide by count"}}{\underbrace{\frac{1}{m}}}\underset{\text{"add them all"}}{\underbrace{\sum _{i=1}^m{x}_i}}$

Reading it term by term: ${\sum }_{i=1}^m{x}_i$ is a compact instruction — "start at the first value $x_1$, keep adding until the last value $x_m$." The $\frac{1}{m}$ out front turns that total into an average. For our batch: $\frac{2+4+4+6+9}{5}=\frac{25}{5}=5$.

WHY. Before we can center the cloud on zero, we must know where its center currently is. ${\mu }_{\mathcal{B}}$ is that "where." Why the mean and not, say, the median? Because the next steps (variance, and the gradient) are built from sums of squares, and the mean is the point that minimizes total squared distance — it is the natural companion to variance.

PICTURE. The yellow tick is the balance point ${\mu }_{\mathcal{B}}=5$. Imagine the number line as a see-saw with a dot's weight at each value: it balances exactly under the yellow tick.

---

Step 3 — Measure the spread (the variance ${\sigma }_{\mathcal{B}}^2$)

WHAT. For each dot, measure its distance from the center, square that distance, then average the squares. That average is the variance; its square root is the standard deviation ${\sigma }_{\mathcal{B}}$ — a single number for "typical width of the cloud."

${\sigma }_{\mathcal{B}}^2=\frac{1}{m}{\sum }_{i=1}^m\underset{\text{squared distance of dot }i\text{ from center}}{\underbrace{(x_i-{\mu }_{\mathcal{B}}{)}^2}}$

Term by term: $(x_i-{\mu }_{\mathcal{B}})$ is the signed gap of dot $i$ from the yellow tick — negative if it sits left of center, positive if right. Squaring does two jobs: it kills the sign (a gap of $-3$ and $+3$ are equally "far"), and it punishes far-away dots more. Averaging with $\frac{1}{m}$ gives one spread number.

For our batch, gaps are $[-3,-1,-1,+1,+4]$, squares $[9,1,1,1,16]$, sum $=28$, so ${\sigma }_{\mathcal{B}}^2=\frac{28}{5}=5.6$ and ${\sigma }_{\mathcal{B}}=\sqrt{5.6}\approx 2.366$.

WHY. To shrink the cloud to a standard width, we must first know its current width. Variance is that measurement. Why divide by $m$ and not $m-1$? BN deliberately uses the biased ($/m$) estimator — it treats the batch as the whole world for this step, matching the parent note's formula.

PICTURE. Each red segment is a squared distance drawn as a little square whose side is the gap. The variance is the average area of those squares.

---

Step 4 — Standardize (subtract, then divide → ${\hat{x}}_i$)

WHAT. Slide the whole cloud left by ${\mu }_{\mathcal{B}}$ so its center lands on $0$; then squeeze it by dividing every number by $\sqrt{{\sigma }_{\mathcal{B}}^2+ϵ}$ so its width becomes $1$.

${\hat{x}}_i=\frac{\stackrel{\text{recenter: center}\to 0}{\overbrace{x_i-{\mu }_{\mathcal{B}}}}}{\underset{\text{rescale: width}\to 1}{\underbrace{\sqrt{{\sigma }_{\mathcal{B}}^2+ϵ}}}}$

The numerator moves the cloud so its middle is at zero (this is the slide). The denominator is (almost) the standard deviation, so dividing by it makes the typical width equal to $1$ (this is the squeeze). The little $ϵ$ (a tiny number like $10^{-5}$) is a safety cushion — we meet it properly in Step 6.

For our batch, gaps $[-3,-1,-1,1,4]$ divided by $\sqrt{5.6}\approx 2.366$ give $\hat{x}\approx [-1.268,-0.423,-0.423,0.423,1.690].$ Check: these average to $0$ and have variance $1$. ✓

WHY. Now every neuron, in every layer, hands the next layer a cloud with the same center and the same width, no matter how the layers below drift. The next layer stops chasing a moving target — the core promise of BN, and why we can use larger, safer learning rates.

PICTURE. Two arrows tell the whole story: a yellow slide arrow moving the cloud onto $0$, and a green squeeze bracket pulling the edges inward to width $1$.

---

Step 5 — Give the freedom back (scale $\gamma$, shift $\beta$ → $y_i$)

WHAT. After forcing center $0$, width $1$, we let the network re-choose its own center and width using two learnable knobs.

$y_i=\underset{\text{"gain": new width}}{\underbrace{\gamma }}{\hat{x}}_i+\underset{\text{"shift": new center}}{\underbrace{\beta }}$

$\gamma$ multiplies, so it sets the new width (a stretch if $|\gamma |>1$, a shrink if $|\gamma |<1$). $\beta$ adds, so it sets the new center. Both are learned by gradient descent, exactly like weights — the network dials them to whatever it finds useful.

With $\gamma =2,\beta =1$: $y=2\hat{x}+1\approx [-1.536,0.155,0.155,1.845,4.380]$.

WHY. Forcing mean $0$/var $1$ can cost the network expressive power — a $\tanh$ neuron, for instance, needs inputs away from $0$ to behave nonlinearly. So we don't leave the network stuck at the standardized cloud; we hand it two dials. Crucially, if identity were best, it could set $\gamma =\sqrt{{\sigma }_{\mathcal{B}}^2+ϵ}$ and $\beta ={\mu }_{\mathcal{B}}$ to recover the original numbers. So BN never removes power — it only changes the starting point the network optimizes from.

PICTURE. The standardized cloud (from Step 4) sits at center $0$, width $1$; the $\gamma ,\beta$ dials stretch and slide it to a chosen place. The gradients that train these dials come from Backpropagation.

---

Step 6 — Edge case: the constant batch (${\sigma }_{\mathcal{B}}^2=0$)

WHAT. What if every number in the batch is identical, e.g. $x=[7,7,7]$? Then every gap is $0$, so ${\sigma }_{\mathcal{B}}^2=0$, and the Step-4 denominator would be $\sqrt{0}=0$ — a division by zero.

${\hat{x}}_i=\frac{7-7}{\sqrt{0+ϵ}}=\frac{0}{\sqrt{ϵ}}=0.$

The $ϵ$ in the denominator is what saves us: instead of $\frac{0}{0}$ (undefined) we get $\frac{0}{\text{tiny}}=0$, a clean answer.

WHY. Real batches sometimes are constant (a dead ReLU, a saturated region). Without $ϵ$ the whole forward pass would produce NaN and training would collapse. With it, a constant batch maps to all-zeros — meaning BN erases information that is constant across the batch. That is precisely why tiny batches hurt BN: with few examples, the mean/variance estimates are noisy or near-degenerate. In that regime prefer Layer Normalization or Group Normalization, which don't rely on a batch of examples.

PICTURE. All three dots stack on top of each other at $7$; the width is literally zero, the yellow center sits right on them, and the output collapses to $0$.

---

Step 7 — Edge case: one example at test time (running stats)

WHAT. At inference you may get a single input — there is no batch to average over, so ${\mu }_{\mathcal{B}},{\sigma }_{\mathcal{B}}^2$ don't exist. Instead, during training BN quietly accumulates a running average of the batch statistics, and uses those fixed numbers at test time.

${\mu }_{run}\leftarrow (1-\alpha ){\mu }_{run}+\alpha {\mu }_{\mathcal{B}},{\sigma }_{run}^2\leftarrow (1-\alpha ){\sigma }_{run}^2+\alpha {\sigma }_{\mathcal{B}}^2$

Here $\alpha$ (a small number like $0.1$) is the update rate: each new batch nudges the running estimate a little toward the batch's own statistic, so ${\mu }_{run}$ becomes a smoothed picture of the whole training set. At test time:

$y=\gamma \frac{x-{\mu }_{run}}{\sqrt{{\sigma }_{run}^2+ϵ}}+\beta .$

Worked check. With ${\mu }_{run}=3.0,{\sigma }_{run}^2=4.0,\gamma =2,\beta =1,ϵ=0$ and a single new input $x=5$: $\hat{x}=\frac{5-3}{\sqrt{4}}=1$, so $y=2(1)+1=3$.

WHY. Test outputs must be deterministic — the same $x$ must give the same $y$ regardless of who else is (or isn't) in the batch. Running statistics guarantee that. A bonus: since ${\mu }_{run},{\sigma }_{run}^2,\gamma ,\beta$ are now all fixed constants, the whole BN operation becomes one straight-line (affine) map that can be folded into the previous layer's weights — zero extra cost at inference.

PICTURE. A timeline of training batches, each nudging a slowly-moving running mean toward the true population center; at test the single dot is standardized by that frozen value.

---

The one-picture summary

Read this figure left to right: raw cloud → measure center → measure width → slide to $0$ → squeeze to width $1$ → apply learnable $\gamma ,\beta$. Every arrow is one of the seven steps above.

Recall Feynman retelling of the whole walkthrough

You have a spilled handful of beads on a ruler. First you find the point where the ruler would balance — that's the mean. Then you measure how far the beads sprawl out on average — that's the spread. Now you do two clean moves: slide the whole handful so the balance point sits on the zero mark, and squeeze it so the sprawl becomes exactly one ruler-unit wide. Every neuron in the network now hands over beads arranged the same tidy way, so nobody downstream is surprised. Finally you give the network two dials — one to stretch the beads back out to whatever width it likes, one to slide them to whatever center it likes — because sometimes tidy-at-zero isn't what it wants, but now it starts from tidy instead of chaos. Two special cases: if all the beads landed on the exact same spot (zero spread) a tiny safety number keeps you from dividing by zero and the output is just zero. And when a single bead shows up at test time with no handful to average, you use a running memory of where past handfuls balanced and how wide they were — so the answer is always the same for the same bead.

---

Related: Training Deep Networks · Weight Initialization · Dropout · 3.2.08 Batch normalization (Hinglish)