Foundations — Data parallelism and ZeRO optimization

Before you can read the parent note Data Parallelism and ZeRO, you need to earn every symbol it throws at you. This page builds each one from nothing, in an order where each idea leans on the one before it. Nothing here assumes you have seen distributed training before.

1. What is a "model" and what is a "parameter"?

Picture a giant control panel covered in millions of little dials. Each dial is one number. Turning the dials is training. A modern model can have hundreds of billions of dials.

Why does the topic need $\Phi$? Because everything about memory is measured in "how many bytes per parameter." If we know $\Phi$ and know how many bytes each dial costs, we can compute the total memory. $\Phi$ is the ruler the whole topic measures with.

Look at the figure: the same panel of dials is copied onto GPU 0 and GPU 1. That duplication is the villain of the entire topic.

2. Bytes — how memory is actually counted

Two number formats matter here:

Why does the topic care? Because memory = (number of dials) × (bytes per dial). A dial stored in fp16 costs $2$ bytes, so all $\Phi$ parameters in fp16 cost $2\Phi$ bytes. That is exactly where the parent's $2\Phi$ comes from. This precision trade-off is explored fully in Mixed Precision Training.

3. Gradient — the "which way to turn each dial" arrow

Because there is exactly one gradient number per parameter, gradients also take $\Phi$ numbers → in fp16 that is another $2\Phi$ bytes. That is the parent's second $2\Phi$ term.

The symbol:

4. $\theta$, $\mathcal{L}$, and the learning rate $\eta$

Why $\eta$ must be small, and why that matters later: tiny steps mean the change to a dial can be a very small number. If we stored dials in fp16, such a tiny change might round to zero and get lost. That is the whole reason the topic keeps an fp32 "master copy" — see §7.

5. The optimizer and its extra memory (Adam)

To be smart, Adam stores two extra numbers per parameter:

Both are kept in fp32 (4 bytes each) for stability. Plus a fp32 master copy of the parameters. So Adam's bookkeeping costs:

This is the parent's mysterious $12\Phi$. More frugal versions are covered in Memory-Efficient Optimizers. Adding it all up gives the famous $16\Phi$:

$M_{\text{gpu}}=\underset{\text{params}}{\underbrace{2\Phi }}+\underset{\text{grads}}{\underbrace{2\Phi }}+\underset{\text{optimizer}}{\underbrace{12\Phi }}=16\Phi \text{ bytes}$

The bar chart shows why the optimizer states (the tall $12\Phi$ block), not the parameters, dominate memory — this is exactly what ZeRO attacks first.

6. GPU, rank, and $N$ — the cast of machines

Why the topic needs these: the entire memory-saving trick is "divide by $N$." Every time you see $\frac{1}{N}$ in the parent note, it literally means "each of the $N$ GPUs keeps only its share." And $r$ lets us say which share belongs to which GPU.

7. Why an fp32 master copy exists (the underflow picture)

Recall from §4 that a step is $\eta \times \text{gradient}$, often a minuscule number. In fp16 the smallest gaps between representable numbers are large; a tiny update lands between two fp16 values and rounds away to nothing — the dial never moves. Keeping the master weights in fp32 (finer gaps) lets those tiny nudges accumulate. This is why $12\Phi$ includes a full fp32 copy. Full detail lives in Mixed Precision Training.

8. Talking between GPUs — collective operations

The GPUs must share their gradient arrows. The messaging patterns are named "collectives." Full treatment is in AllReduce and Collective Communication; here are just the three the parent uses.

The averaging formula the parent writes, ${\bar{\nabla }}_{\theta }=\frac{1}{N}{\sum }_{r=1}^N{\nabla }_{{\theta }_r}\mathcal{L}({\mathcal{B}}_r),$ is just AllReduce in symbols: ${\sum }_{r=1}^N$ means "add up over all GPUs $r$," and $\frac{1}{N}$ averages them. It works because the gradient of the whole dataset equals the average of gradients over its parts.

9. Partitioning — the one word that IS ZeRO

This is why the reduction factor climbs toward $N\times$: cut everything into $N$ pieces, hold one piece each.

Prerequisite map

Related deeper topics that build on this same foundation: Gradient Accumulation, Activation Checkpointing, Pipeline Parallelism, and the big picture in Large Language Models Training.

Equipment checklist

Cover the right side and test yourself. If any answer surprises you, re-read that section above.