Question bank — Batch normalization

False — during training BN uses only the current mini-batch's mean and variance; the whole-set (running) statistics are reserved for inference.

False — they can recover the original scale, but they are learned, and normalization already reshaped the loss landscape and gradient flow before they act. See Optimization and Conditioning of the Loss Surface.

True — if you multiply that layer's weights by a constant, the pre-activation scales by the same constant, but BN's division by $\sqrt{{\sigma }^2+ϵ}$ cancels it, so the normalized output is unchanged.

False — with fixed running $\mu$ and ${\sigma }^2$, BN reduces to a single affine map $y=ax+b$, which can even be folded into the previous layer's weights.

False (redundant) — BN subtracts the mean, cancelling any constant $b$; the $\beta$ parameter already plays the bias role, so the earlier bias just wastes a parameter.

False — it reduces the shift in each layer's input distribution but cannot eliminate it, since the running statistics still drift as training proceeds.

True — because each example is normalized using its random batch-mates, the same input gets slightly different normalized values across batches; this noise acts like a mild regularizer, sometimes reducing the need for Dropout.

False — a batch of one has variance $0$, so $\hat{x}=0/\sqrt{ϵ}\approx 0$ for every input; BN collapses all information and is effectively useless.

True — the forward normalization divides the summed squared deviations by $m$; the unbiased correction is a separate detail used only for the running-variance estimate in some implementations.

False — BN makes training far less sensitive to initialization, but pathological inits can still cause problems in un-normalized paths; see Weight Initialization.

It confuses $\hat{x}$ with $y$; the output $y=\gamma \hat{x}+\beta$ has mean $\beta$ and variance ${\gamma }^2$, which the network learns and which is generally not $0$/$1$.

$ϵ$ is there to prevent division by zero (and stabilize near-zero variance), not to inflate values; it is tiny (e.g. $10^{-5}$) precisely so it barely changes normal cases.

You divide by the standard deviation $\sqrt{{\sigma }^2+ϵ}$, not the variance; dividing by variance would give wrong units and would not yield unit variance.

The original scheme places BN between the linear map $Wx$ and the nonlinearity; the choice is a design decision, not a fixed law, so "always after" is an overstatement.

That would make outputs depend on which other examples happen to share the batch, breaking determinism; inference uses the fixed running statistics learned during training.

$\beta$ is shared across all $m$ examples in the batch, so its gradient is the sum ${\sum }_i\partial L/\partial y_i$, not a single term. See Backpropagation.

It adds two learnable parameters per feature, $\gamma$ and $\beta$; they are few, but they are real trainable parameters.

By keeping each layer's inputs well-scaled and better-conditioned, gradients point more usefully, so larger steps don't blow activations up or land in saturated regions.

Early in training the population statistics are unknown and shifting; per-batch statistics adapt automatically and their noise even helps regularize, so freezing too early would hurt both accuracy and stability.

Normalizing keeps pre-activations near a moderate range instead of drifting into the flat saturated tails, where gradients would shrink toward zero. See Vanishing and Exploding Gradients.

Because $\gamma$ and $\beta$ can restore any mean and scale — in particular setting $\gamma =\sqrt{{\sigma }^2+ϵ}$, $\beta =\mu$ recovers the original activation exactly, so nothing is permanently lost.

The stochastic normalization jitters each activation slightly, acting like injected noise that discourages over-reliance on any single unit — a regularizing effect.

With fixed running statistics it is a constant affine transform; composing two affine maps gives another affine map, so it merges into $W,b$ with zero extra runtime cost.

All deviations are zero so ${\sigma }^2=0$ and each ${\hat{x}}_i=0$; the constant-across-batch information is erased, and only the $ϵ$ prevents a $0/0$.

The mean/variance estimates become noisy and unreliable, degrading accuracy; in this regime use Layer Normalization or Group Normalization, which do not depend on batch size.

You get a genuine $0/0$ division — undefined or NaN — which is exactly the failure $ϵ$ exists to prevent.

Its normalized output is squashed to nearly the constant $\beta$, so the unit contributes almost no input-dependent signal — a soft way for the network to switch a feature off.

They are still at their initial values (often mean 0, variance 1) and have barely absorbed any batch statistics, so early-training inference via running stats is unreliable until enough updates accumulate.

Every layer's output is forced to mean 0 / variance 1, which may clash with what a nonlinearity needs; the network loses the freedom to pick its preferred scale, potentially limiting expressiveness.