3.3.9 · D5Deep Learning Frameworks
Question bank — Mixed precision training
Two words you must have crisp before starting:
- Underflow — a number too small to represent becomes exactly . (FP16 loses everything below about .)
- Overflow — a number too large becomes
inf(infinity). (FP16 tops out near .)
Loss scaling fights underflow; it can cause overflow. Almost every trap below is really a question about which of those two you are trading against.
True or false — justify
FP16 is always faster than FP32 on any GPU
False — the speedup comes from dedicated Tensor Cores; on a GPU without them FP16 can be no faster or even slower, since it just emulates the ops while paying extra cast overhead.
Mixed precision reduces accuracy of the final trained model
False — done correctly (FP32 master weights + loss scaling) final accuracy matches FP32 training, because every decision-critical accumulation still happens in FP32; only the throwaway forward/backward math is FP16.
The FP32 master weights are needed because FP16 can't represent the weight values themselves
False — a weight like is perfectly representable in FP16. The master copy exists because tiny updates (e.g. ) are lost when added to a larger weight in FP16; it protects the accumulation, not the storage. See 3.3.08-Gradient-accumulation for the same "small-adds-into-big" hazard.
Loss scaling changes what the network learns
False — multiplying the loss by scales every gradient by the same , and we divide it back out before the optimizer step; the update direction and magnitude are identical, we only borrowed range temporarily.
BF16 needs loss scaling just like FP16 does
False in practice — BF16 keeps FP32's 8 exponent bits, so its dynamic range reaches ; gradients rarely underflow, so loss scaling is usually unnecessary. Its cost is fewer mantissa bits (coarser precision).
A larger loss scale is always safer
False — larger pushes small gradients out of underflow but pushes large gradients toward
inf; too big an causes overflow and produces nan. Safety is a two-sided window, not "bigger is better".You can drop the FP32 master weights if you use dynamic loss scaling
False — these solve different problems. Loss scaling fixes gradient underflow during backprop; the FP32 master fixes update underflow during the optimizer step. Removing either reintroduces its own failure.
If gradients never overflow, dynamic loss scaling will keep constant forever
False — after stable steps it doubles to reclaim range for the smallest gradients, then backs off only when an overflow finally appears; it deliberately probes upward.
Spot the error
"We got nan in the loss, so our learning rate is too high."
Not necessarily — with mixed precision
nan most often means the loss scale overflowed gradients to inf, which then became nan. First halve before touching the learning rate; the LR may be fine."To save memory we store the Adam optimizer moments in FP16."
Dangerous — the optimizer's running variance/mean accumulate over thousands of steps, exactly the cumulative-tiny-add situation FP16 corrupts. Optimizer state normally stays FP32; that's why Example 3's optimizer state was the un-halved item.
"After backprop we update the master weights, then unscale the gradients."
Wrong order — you must unscale first, otherwise you apply an update that is too large. Unscaling must happen between backward and
optimizer.step."We scaled each layer's gradient by individually to guarantee coverage."
Redundant and wrong — scaling the single scalar loss at the top of the graph makes the chain rule carry the factor into every gradient automatically. Per-layer scaling is extra work and can double-scale.
"An overflow happened this step, so we halve and apply the (now smaller) gradients anyway."
Wrong — an overflowed step contains
inf/nan gradients that are garbage. You must skip the update entirely for that step, halve , and redo; applying them corrupts the weights."We use FP16 for batch-norm statistics to save time."
Risky — BN computes means and variances by summing over the batch, and those reductions lose precision badly in FP16. Reductions/normalization layers are typically kept in FP32 even inside mixed precision.
"BF16 has the same range as FP32, so it also has the same precision."
Wrong — same range (8 exponent bits) but fewer mantissa bits (7 vs 23), so BF16 is much coarser; it trades precision to keep range.
Why questions
Why scale the loss rather than scale the gradients after they're computed?
Because by the time a gradient is computed in FP16 it has already underflowed to zero — there's nothing left to scale up. Scaling the loss injects the factor before backprop so every gradient is born pre-shifted into range.
Why keep two weight copies instead of just doing the whole update in FP16?
Because the update is often smaller than FP16's precision step (); adding it to a larger FP16 weight rounds to no change. FP32 has fine enough steps () to let tiny updates accumulate.
Why do early layers underflow more than late layers?
The chain rule multiplies many small factors as gradients flow backward, so gradients shrink deeper into the network; early-layer gradients (near –) sit closest to FP16's floor and vanish first.
Why is halving/doubling used for dynamic instead of adding a constant?
Because floating-point range is multiplicative (powers of 2 in the exponent); multiplying by 2 shifts gradients by exactly one exponent step, matching how the format actually works.
Why does mixed precision help distributed training beyond just compute?
Gradients must be communicated across devices to sync; FP16 halves the bytes on the wire, cutting bandwidth — a real bottleneck in model/data parallelism.
Why do activations dominate the memory savings, not weights?
Because activations scale with batch size × sequence length and are stored for every layer during backprop, so they're usually the largest tensor group — halving them (FP16) frees more than halving the fixed-size weights. This is also why gradient checkpointing targets activations.
Edge cases
What happens with a gradient of exactly zero under loss scaling?
Nothing changes — and . A true zero gradient is already representable and carries no update; loss scaling only rescues small nonzero values from rounding to zero.
What if the loss scale shrinks toward and still overflows every step?
That signals genuinely exploding gradients, not a precision artifact — the fix is now an optimization fix (lower LR, gradient clipping), because even unscaled the gradients are too large. See 1.3.07-Numerical-stability.
A weight lands exactly on FP16's max (); next update pushes it higher — what happens?
It overflows to
inf on the next FP16 cast. This is rare for weights but is why activations passing through unbounded ops (like large sums) are watched; keeping such ops in FP32 avoids it.The gradient is below FP16's subnormal floor even after scaling by max stable — then?
It still underflows to zero; loss scaling has a ceiling ( can't grow without overflowing large gradients). This is a case where BF16's wider range is the correct tool instead of FP16 + scaling.
First training step: master weights exist but no gradients yet — is loss scaling active?
Yes conceptually, but with dynamic scaling starts at its large initial value and hasn't been tested; the first backward is what reveals whether that overflows, triggering the first halving if needed.
What if two GPUs pick different dynamic values?
They'd unscale by different factors and desynchronise the model — so (and overflow decisions) must be shared/reduced across all workers so every replica skips or applies the same step.
Recall Quick self-test
Name the two distinct underflow problems and their two distinct fixes ::: (1) Gradient underflow in backprop → fixed by loss scaling; (2) Update underflow in the optimizer → fixed by FP32 master weights.
One sentence: what does dynamic loss scaling trade off? ::: It walks upward to rescue small gradients from underflow while backing off whenever large gradients threaten inf overflow.