Batch normalization: Worked examples — AI-ML deep dive

Every symbol we use is already earned in the parent note, but we restate each the moment it appears so you can read this page from line one.

The scenario matrix

Below is every class of situation BN can produce. Each row is a "cell". The worked examples that follow are labelled with the cell(s) they cover, so you can see the whole territory is filled.

#	Case class	What makes it tricky	Covered by
A	Ordinary batch (varied values)	baseline; get the mechanics right	Ex 1
B	Negative & mixed-sign inputs	mean can be negative; deviations flip sign	Ex 2
C	Constant / degenerate batch ( $σ^{2} = 0$ )	division by zero → $ϵ$ saves us	Ex 3
D	Single-element batch ( $m = 1$ )	variance $= 0$ always; BN erases everything	Ex 3 (limit)
E	Large-magnitude / drifting batch	shows why BN exists (saturation)	Ex 4 (figure)
F	Learnable $γ, β$ recover identity	the "BN is pointless?" objection, answered numerically	Ex 5
G	Inference on one example (running stats)	no batch — use $μ_{r u n}, σ_{r u n}^{2}$	Ex 6
H	Backprop numbers ( $\partial L / \partial γ, \partial L / \partial β$ )	gradients must actually be computable	Ex 7
I	Word problem (bias is redundant)	real design decision: drop the bias	Ex 8
J	Exam twist (multi-feature batch, per-feature stats)	BN normalizes each feature separately	Ex 9

Two symbols we lean on constantly:

$m$ = number of examples in the mini-batch $B = {x_{1}, \dots, x_{m}}$ (one scalar activation each).
$μ_{B} = \frac{1}{m} \sum_{i} x_{i}$ (the batch mean, the center), $σ_{B}^{2} = \frac{1}{m} \sum_{i} (x_{i} - μ_{B})^{2}$ (the batch variance, the spread). We divide by $m$ , not $m - 1$ — BN uses the biased estimator.

Ex 1 — Cell A: the ordinary batch

Worked example Baseline: normalize

$x = [2, 4, 6, 8]$ , with $γ = 1, β = 0, ϵ = 0$ Forecast: before reading on — the values are evenly spaced. Guess: will the normalized values also be evenly spaced? Will their mean be exactly 0?

Batch mean $μ_{B} = \frac{2 + 4 + 6 + 8}{4} = 5$ . Why this step? We must know the center before we can subtract it — normalization means "measure each point relative to the center."
Deviations $x_{i} - μ_{B} = [- 3, - 1, 1, 3]$ . Why? These are the raw distances from center; their signs tell us who is below (−) and above (+) the mean.
Variance $σ_{B}^{2} = \frac{9 + 1 + 1 + 9}{4} = 5$ , so $σ_{B} = 5 \approx 2.236$ . Why? We square so below/above both count as "spread" (no cancellation), then average.
Normalize $\overset{x}{^}_{i} = \frac{x _{i} - μ _{B}}{σ _{B}^{2}} = [- 1.342, - 0.447, 0.447, 1.342]$ . Why? Dividing by the spread rescales so the typical distance is 1 — that is exactly what "unit variance" means.
Scale/shift with $γ = 1, β = 0$ : $y_{i} = \overset{x}{^}_{i}$ (unchanged).

Verify: mean of $\overset{x}{^}$ is $\frac{- 1.342 - 0.447 + 0.447 + 1.342}{4} = 0$ ✓, variance $= \frac{1.8 + 0.2 + 0.2 + 1.8}{4} = 1$ ✓. And yes — evenly-spaced input → evenly-spaced output (BN is affine, it never bends the spacing).

Ex 2 — Cell B: negative and mixed-sign inputs

Worked example Mixed signs:

$x = [- 4, 0, 0, 8]$ , $γ = 1, β = 0, ϵ = 0$ Forecast: the mean here is positive even though two entries are negative-or-zero. Guess the mean and whether the two zeros normalize to the same value.

Mean $μ_{B} = \frac{- 4 + 0 + 0 + 8}{4} = 1$ . Why this step? Sign of the mean depends on the whole batch, not on any single entry — you cannot eyeball it from "there are negatives."
Deviations $[- 4 - 1, 0 - 1, 0 - 1, 8 - 1] = [- 5, - 1, - 1, 7]$ . Why? Subtracting a positive center pushes even the zeros to a negative deviation — the two zeros are below average.
Variance $σ_{B}^{2} = \frac{25 + 1 + 1 + 49}{4} = 19$ , $σ_{B} = 19 \approx 4.359$ .
Normalize $\overset{x}{^} = [\frac{- 5}{19}, \frac{- 1}{19}, \frac{- 1}{19}, \frac{7}{19}] \approx [- 1.147, - 0.229, - 0.229, 1.606]$ . Why? Equal inputs (the two zeros) map to equal outputs — BN preserves ties.

Verify: mean of $\overset{x}{^} = \frac{- 1.147 - 0.229 - 0.229 + 1.606}{4} \approx 0$ ✓. Notice the raw zero became a negative normalized value — BN measures "distance from the batch center," not distance from the number line's zero.

Ex 3 — Cells C & D: constant batch and single element (degenerate)

Worked example Degenerate spread:

$x = [5, 5, 5, 5]$ , $ϵ = 1 0^{- 5}$ Forecast: all four values are identical. What is $σ_{B}^{2}$ ? What happens when we divide by $σ_{B}^{2} + ϵ$ if $ϵ$ were $0$ ? Guess $\overset{x}{^}$ .

Mean $μ_{B} = 5$ . Deviations all $0$ . Why? Every value equals the mean, so nothing is "off-center."
Variance $σ_{B}^{2} = 0$ . Why this matters? With $ϵ = 0$ the normalize step would be $\frac{0}{0} = \frac{0}{0}$ — undefined. This is the degenerate case.
Normalize with $ϵ$ : $\overset{x}{^}_{i} = \frac{0}{0 + 1 0 ^{- 5}} = 0$ for all $i$ . Why the $ϵ$ ? It replaces the illegal $\frac{0}{0}$ with a legal $\frac{0}{tiny} = 0$ . The $ϵ$ is a floor on the denominator.

Verify: every output is exactly $0$ ✓ — BN has erased the constant. Whatever information the value $5$ carried is gone (only $β$ could re-inject a constant afterward).

Limiting case D ( $m = 1$ ): with a single example, its own value is the mean, so $σ_{B}^{2} = 0$ always → same degeneracy → output always $0$ (before $β$ ). This is precisely why tiny batches break BN, and why Layer Normalization / Group Normalization (which normalize across features, not across the batch) are used when $m$ is small.

Ex 4 — Cell E: why BN exists (saturation of a drifting batch)

Worked example Large-magnitude batch feeding a sigmoid

A layer's activations have drifted to $x = [10, 12, 14, 16]$ and are fed into a sigmoid $σ (x) = \frac{1}{1 + e ^{- x}}$ . Show why this is a problem, and how BN fixes it.

Forecast: guess $σ (10)$ and $σ (16)$ . How different are they? What does that say about the gradient?

Raw sigmoid outputs: $σ (10) \approx 0.99995$ , $σ (16) \approx 0.9999999$ . Why this step? When inputs are far to the right, sigmoid is flat — outputs are nearly identical no matter the input.
Gradient there $σ^{'} (x) = σ (x) (1 - σ (x))$ : $σ^{'} (10) \approx 4.5 \times 1 0^{- 5}$ , $σ^{'} (16) \approx 1.1 \times 1 0^{- 7}$ . Why? A near-zero derivative means backprop passes almost no signal — this is the vanishing gradient caused by saturation.
Apply BN first ( $γ = 1, β = 0, ϵ = 0$ ): $μ_{B} = 13, σ_{B}^{2} = 5, σ_{B} = 5$ . Then $\overset{x}{^} = [\frac{- 3}{5}, \frac{- 1}{5}, \frac{1}{5}, \frac{3}{5}] \approx [- 1.342, - 0.447, 0.447, 1.342]$ . Why? Now the inputs straddle $0$ , landing in sigmoid's steep region.
Sigmoid after BN: $σ (- 1.342) \approx 0.207, σ (1.342) \approx 0.793$ — clearly distinct, with derivatives near their max $0.25$ .

Verify: the red-highlighted post-BN points in the figure sit on the steep slope; the black pre-BN points sit on the flat plateau where $σ^{'} \approx 0$ . BN moved the batch from "no gradient" to "healthy gradient." ✓ This is the whole reason BN speeds up training — see Optimization and Conditioning of the Loss Surface.

Ex 5 — Cell F: $γ, β$ can recover identity (answering the objection)

Worked example "Isn't BN pointless because

$γ, β$ undo it?" — a numerical answer Take $x = [2, 4, 6, 8]$ again ( $μ_{B} = 5, σ_{B}^{2} = 5$ , $ϵ = 0$ ). Choose $γ = σ_{B}^{2} + ϵ = 5$ and $β = μ_{B} = 5$ . Show the output equals the input.

Forecast: guess the output $y$ . Will it be the original $x$ ?

Normalize: $\overset{x}{^} = [- 1.342, - 0.447, 0.447, 1.342]$ (from Ex 1).
Scale/shift: $y_{i} = 5 \cdot \overset{x}{^}_{i} + 5$ . Why this choice of $γ, β$ ? We deliberately pick them to invert the normalize step — $γ$ re-injects the spread, $β$ re-injects the mean.
Compute: $y_{1} = 5 (- 1.342) + 5 \approx - 3 + 5 = 2$ , and likewise $y = [2, 4, 6, 8]$ .

Verify: $y = x$ exactly ✓. So BN can represent the identity — it never reduces representational power. But the key point (see the Batch normalization parent's Mistake #2): in practice $γ, β$ are learned, and even when the output is shifted back, the gradient flow and conditioning the network trains under have already been changed. Not pointless — free.

Ex 6 — Cell G: inference on a single example

Worked example Test time, one input, no batch

Training left $μ_{r u n} = 6.0, σ_{r u n}^{2} = 9.0, γ = 3, β = - 2, ϵ = 0$ . A new single input $x = 12$ arrives. Compute $y$ .

Forecast: at test time there is no batch — which mean/variance do we use? Guess $y$ .

Use running stats (not a batch mean): $\overset{x}{^} = \frac{x - μ _{r u n}}{σ _{r u n}^{2} + ϵ} = \frac{12 - 6}{9} = \frac{6}{3} = 2$ . Why this step? With one example, a batch mean would be $x$ itself → $\overset{x}{^} = 0$ always (the Cell D degeneracy). Running stats, frozen after training, give a deterministic result independent of what else is in the batch.
Scale/shift: $y = γ \overset{x}{^} + β = 3 (2) + (- 2) = 4$ .

Verify: $y = 4$ ✓. Because $μ_{r u n}, σ_{r u n}^{2}, γ, β$ are all fixed constants, BN at inference is a plain affine map $y = \frac{γ}{σ _{r u n}^{2} + ϵ} x + (β - \frac{γ μ _{r u n}}{σ _{r u n}^{2} + ϵ})$ . Plugging $x = 12$ : slope $= \frac{3}{3} = 1$ , intercept $= - 2 - \frac{3 \cdot 6}{3} = - 8$ , so $y = 1 \cdot 12 - 8 = 4$ ✓ (matches). This is why BN can be folded into the previous layer's weights at deployment for zero extra cost.

Ex 7 — Cell H: backprop numbers through $γ$ and $β$

Worked example Parameter gradients from upstream signal

During backprop a batch of $m = 3$ gives normalized values $\overset{x}{^} = [- 1.0, 0.0, 1.0]$ and upstream gradients $\frac{\partial L}{\partial y} = [0.5, - 0.2, 0.3]$ . Compute $\frac{\partial L}{\partial γ}$ and $\frac{\partial L}{\partial β}$ .

Forecast: the parent note says $\frac{\partial L}{\partial γ} = \sum_{i} \frac{\partial L}{\partial y _{i}} \overset{x}{^}_{i}$ and $\frac{\partial L}{\partial β} = \sum_{i} \frac{\partial L}{\partial y _{i}}$ . Guess both sums.

$\frac{\partial L}{\partial γ}$ : $\sum_{i} \frac{\partial L}{\partial y _{i}} \overset{x}{^}_{i} = 0.5 (- 1.0) + (- 0.2) (0.0) + 0.3 (1.0) = - 0.5 + 0 + 0.3 = - 0.2$ . Why this form? Since $y_{i} = γ \overset{x}{^}_{i} + β$ , we have $\frac{\partial y _{i}}{\partial γ} = \overset{x}{^}_{i}$ . Chain rule: multiply upstream by $\overset{x}{^}_{i}$ , then sum because $γ$ is shared across all $m$ examples (see Backpropagation).
$\frac{\partial L}{\partial β}$ : $\sum_{i} \frac{\partial L}{\partial y _{i}} = 0.5 - 0.2 + 0.3 = 0.6$ . Why? $\frac{\partial y _{i}}{\partial β} = 1$ , so we just sum the upstream gradients.

Verify: $\frac{\partial L}{\partial γ} = - 0.2$ , $\frac{\partial L}{\partial β} = 0.6$ ✓. Sanity: $β$ is a pure additive shift, so its gradient is just the total upstream flow — units and sign both check out.

Ex 8 — Cell I: word problem (the bias is redundant)

Worked example Design decision on a real layer

An engineer builds a layer $z = W x + b$ followed by BN. She sets $b = 3$ (a constant) and worries it interacts badly with BN. Show what BN does to that bias, using a batch of pre-BN values $z = [3 + 1, 3 - 1, 3 + 2, 3 - 2] = [4, 2, 5, 1]$ .

Forecast: the bias adds the same $3$ to every value. What does BN's subtract-the-mean step do to a value that is present in every entry?

Mean $μ_{B} = \frac{4 + 2 + 5 + 1}{4} = 3$ . Why? The bias $b = 3$ shifts every $z_{i}$ up by 3, so it shifts the mean up by exactly 3 too.
Deviations $z_{i} - μ_{B} = [1, - 1, 2, - 2]$ — the $3$ has cancelled. Why? Subtracting the mean removes any component common to all examples; a constant bias is exactly such a component.
Consequence: the same deviations $[1, - 1, 2, - 2]$ arise for any value of $b$ . The bias has zero effect on BN's output.

Verify: repeat with $b = 100$ ( $z = [101, 99, 102, 98]$ ): $μ_{B} = 100$ , deviations $[1, - 1, 2, - 2]$ — identical ✓. Lesson: set bias=False in the layer before BN; the learnable $β$ already supplies the shift. Carrying $b$ wastes parameters and is exactly the parent's Mistake #4.

Ex 9 — Cell J: exam twist (per-feature normalization)

Worked example Two features, normalize each independently

A mini-batch of $m = 3$ examples, each with 2 features, arranged as rows: $X = 135102030$ Feature 1 is the left column, feature 2 the right. Normalize with $γ = [1, 1], β = [0, 0], ϵ = 0$ . Trap: do NOT pool all six numbers together.

Forecast: guess $μ$ for each feature. Will the two features get the same normalized column even though their scales (1–5 vs 10–30) are so different?

Per-feature means (average down each column): feature 1 $μ = \frac{1 + 3 + 5}{3} = 3$ ; feature 2 $μ = \frac{10 + 20 + 30}{3} = 20$ . Why per feature? BN normalizes each activation dimension separately — mixing features would let a large-scale feature dominate a small one. Each feature keeps its own $μ, σ^{2}$ .
Per-feature variance: feature 1: deviations $[- 2, 0, 2]$ , $σ^{2} = \frac{4 + 0 + 4}{3} = \frac{8}{3} \approx 2.667$ ; feature 2: deviations $[- 10, 0, 10]$ , $σ^{2} = \frac{100 + 0 + 100}{3} = \frac{200}{3} \approx 66.667$ .
Normalize each column: feature 1: $\overset{x}{^} = [\frac{- 2}{8/3}, 0, \frac{2}{8/3}] \approx [- 1.225, 0, 1.225]$ ; feature 2: $\overset{x}{^} = [\frac{- 10}{200/3}, 0, \frac{10}{200/3}] \approx [- 1.225, 0, 1.225]$ . Why do they match? Both columns are linearly evenly spaced, so after standardizing each by its own spread they land on the identical pattern — BN has put the two very different scales onto a common footing.

Verify: each normalized column has mean $0$ and variance $1$ (e.g. feature 2: $\frac{1.22 5 ^{2} + 0 + 1.22 5 ^{2}}{3} = 1$ ✓). The whole point: BN made the wildly different scales (1–5 vs 10–30) comparable — this is the "well-conditioned" property that lets you raise the learning rate.

Recall Which cell breaks BN, and what replaces it?

Small-batch / single-example degeneracy (Cells C, D) makes $σ_{B}^{2}$ noisy or zero. ::: Use Layer Normalization or Group Normalization, which normalize across features within one example, so they don't depend on batch size.

See also: Weight Initialization, Dropout, and the parent Batch normalization · Training Deep Networks.