Visual walkthrough — Data parallelism and ZeRO optimization
Before anything, let us agree on the ONE quantity everything is built from.
Step 1 — One number is not one number: the bytes behind a parameter
WHAT. We look at a single parameter during training and ask: how many bytes of RAM does it actually cost? The surprising answer is not 2 or 4 — it is 16.
WHY. Modern training (mixed precision + the Adam optimizer) keeps several copies and helpers for every parameter. If we don't count these first, every later formula is a mystery. This is the atom of the whole derivation — see Mixed Precision Training and Memory-Efficient Optimizers for why each copy exists.
PICTURE. Look at the stacked bar. One parameter drags along a whole entourage.
Group these into two families, because ZeRO treats them differently:
The block (momentum + variance + master weight ) is the fat part. Remember it — it gets attacked first.
Step 2 — Data parallelism: everybody carries the same suitcase
WHAT. We put GPUs to work. Standard data parallelism gives each GPU a complete copy of the model state. They differ only in which data they see.
WHY. Copying the whole model onto every GPU is what lets each one run a full forward/backward pass alone, without asking anyone else for a weight. Simple — but every GPU stores the identical . That is pure redundancy.
PICTURE. Four identical suitcases, one per GPU. Different data batches () flow in, but the model bytes are clones.
Step 3 — ZeRO-1: cut the fattest block into N slices
WHAT. Notice the optimizer block is the biggest single piece. ZeRO-1 keeps params () and grads () replicated, but partitions the : GPU keeps optimizer states for only its slice .
WHY. The optimizer state for a parameter depends only on that parameter's own gradient history — momentum for weight #5 needs only gradient #5. So GPU can update its slice with its local optimizer state and nobody else's. No information is lost by splitting. After the update, an AllGather reassembles the full parameter vector on every GPU.
PICTURE. The bar gets chopped into colored slices, one live on each GPU; the grey slices are "not stored here".
The reduction factor versus : For : .
Step 4 — ZeRO-2: slice the gradients too
WHAT. Now also partition the gradients. GPU ends up holding only the averaged gradient for its own slice .
WHY. Look at Step 3: GPU only updates its slice , so it only needs the averaged gradient for . Storing the other GPUs' gradient slices is dead weight the moment the backward pass finishes. The tool that delivers "average and hand each GPU only its slice" in one shot is ReduceScatter — think of AllReduce and a partition fused together.
PICTURE. Each GPU computes all gradients, then ReduceScatter throws away everyone else's slice and keeps only the summed-and-averaged own slice.
Step 5 — ZeRO-3: slice the parameters — the full
WHAT. Partition the final block, the parameters. GPU permanently stores only of the weights. Every memory component is now sliced.
WHY. A GPU rarely needs all layers at once — it processes layer-by-layer. So we keep only our own slice resting in memory. When we reach a layer whose weights live elsewhere, we AllGather them just-in-time, compute, then discard them. We pay in communication (gathering weights twice per step, forward and backward) to win the last block of memory. See Activation Checkpointing and Gradient Accumulation for complementary ways to shrink the activation memory this frees room for.
PICTURE. All three blocks now striped into slices; only the on-GPU slice is solid, the rest are ghostly and fetched on demand.
Step 6 — The bill: what ZeRO-3 pays in communication
WHAT. ZeRO-3 is not free. Per step it moves roughly 3× the bytes of plain data parallelism.
WHY. Standard DP does one AllReduce on the gradients. ZeRO-3 does: AllGather params in the forward pass, AllGather params again in the backward pass, and ReduceScatter grads once. Each collective on a size- object moves bytes.
PICTURE. Two timelines side by side — one arrow for standard DP, three for ZeRO-3.
The one-picture summary
One bar of , watched as ZeRO peels it apart:
Recall Feynman retelling — the whole walkthrough in plain words
Every parameter is really 16 bytes of stuff: a fast half-size weight, a matching gradient, and a heavy trio the optimizer needs (momentum, variance, and a careful full-size master copy). So the model costs bytes.
Plain data parallelism hands every GPU a complete suitcase — identical bytes, only the data differs. With 8 GPUs you store 8 identical copies: 7 are pure waste.
ZeRO deletes that waste in three cuts. ZeRO-1 slices the heavy optimizer trio () into pieces — safe, because each parameter's optimizer state only cares about its own gradient. ZeRO-2 also slices the gradients, using ReduceScatter to hand each GPU just the averaged slice it will update. ZeRO-3 slices the weights too, fetching missing weights on the fly with AllGather and throwing them away after use — so every block is divided by , giving per GPU, the full win.
The price is communication: ZeRO-3 moves about three times the bytes of standard DP each step, because it gathers weights in both the forward and backward passes plus scatters the gradients. Fast interconnect makes it worthwhile; slow networks make you reconsider.
Key self-tests: