Foundations — Batch normalization

Before you can read the parent note Batch normalization, you need to own every symbol it throws at you. This page builds each one from absolute zero: plain words → a picture → why the topic needs it.

0. The scene: what is "an activation flowing through a network"?

A neural network is a stack of layers. Each layer takes some numbers, multiplies and adds them, then passes the results up. A single number arriving at one spot in one layer is called an activation.

If you are hazy on how numbers move layer-to-layer, revisit Backpropagation and Training Deep Networks — this page assumes only that "a layer produces numbers".

Look at figure s01: one column of the network is highlighted. The little dots are activations. BN operates on one such dot's value, collected across many examples. That collection is our next symbol.

1. The mini-batch: $\mathcal{B}$, $m$, and the subscript $i$

We rarely feed the network one example at a time. We feed a small group of examples together — a mini-batch — because the average behaviour of a group is smoother and computers process groups efficiently.

Why the topic needs it: BN's whole trick is to look at one fixed position across the whole batch and standardize those $m$ readings together. Without the idea of "a batch of $m$ values from one position", there is nothing to compute statistics over.

2. Summation: the symbol $\sum$

To find the average of the pile we must add all the values up. The Greek capital sigma $\sum$ is shorthand for "add these up".

Why we need it: every statistic below (mean, variance) is secretly one big sum. The symbol saves us from writing "$x_1+x_2+\dots$" forever.

3. The mean ${\mu }_{\mathcal{B}}$ — the center of the pile

Why BN needs it: to center the pile at zero, you first have to know where its center currently is. That center is ${\mu }_{\mathcal{B}}$, and Step 1 of BN subtracts it.

4. Deviation and variance ${\sigma }_{\mathcal{B}}^2$ — the spread of the pile

Knowing the center isn't enough; we also need to know how spread out the values are.

We can't just average the deviations — the positive and negative ones cancel to zero (that's what "center" means). So we square each deviation first (squaring makes everything positive and punishes far-away points more), then average.

Why BN needs it: to make the spread equal to one unit, you must know the current spread. That is ${\sigma }_{\mathcal{B}}$, and Step 3 divides by it.

5. Normalization ${\hat{x}}_i$ and the guard $ϵ$

Now combine center and spread. "Standardize" means: subtract the center, divide by the spread.

Figure s03 shows the transformation: the shape of the pile is unchanged, but it slides to sit over $0$ and squeezes/stretches to width $\approx 1$. That reshaping is the entire point of BN's first three steps.

6. The learnable dials $\gamma$ and $\beta$

Forcing every pile to mean-0/variance-1 might be too rigid — some layers work better with inputs shifted or stretched elsewhere. So BN adds two adjustable knobs the network learns on its own.

Why the topic needs them: without $\gamma ,\beta$, BN would trap every layer at a single distribution shape. These dials restore full freedom while keeping the training benefits.

7. Running averages: ${\mu }_{run}$, ${\sigma }_{run}^2$, $\alpha$

At test time you may get one example — you can't compute a batch mean from a single number. So during training BN quietly keeps a slowly-updated memory of typical statistics.

Why the topic needs it: these frozen ${\mu }_{run},{\sigma }_{run}^2$ make BN deterministic at test — same output no matter what else is in the batch. This is the train-vs-test distinction the parent stresses.

8. The gradient symbol $\frac{\partial L}{\partial (\cdot )}$

BN must be trainable, so we need to know how a change in each quantity changes the loss.

For BN the two key ones are simple because $y_i=\gamma {\hat{x}}_i+\beta$: $\frac{\partial L}{\partial \gamma }={\sum }_i\frac{\partial L}{\partial y_i}{\hat{x}}_i,\frac{\partial L}{\partial \beta }={\sum }_i\frac{\partial L}{\partial y_i}$ Full mechanics live in Backpropagation.

How these feed the topic

Related normalizers that reuse this same vocabulary but pool differently: Layer Normalization and Group Normalization. The problems BN attacks are covered in Vanishing and Exploding Gradients, Weight Initialization, and Optimization and Conditioning of the Loss Surface; its mild regularizing overlap with Dropout is discussed in the parent.

Equipment checklist

Test yourself — reveal only after you've answered aloud.