Question bank — FSDP and sharded training

False — the persistent total is unchanged (one full copy of everything still exists, just split up). FSDP reduces memory per device by $\frac{1}{N}$, which is what lets a giant model fit at all.

False — only the persistent params/grads/optimizer states divide by $N$. The temporary all-gather buffer, the activations, and framework overhead do not shard, so real reduction is less than $N\times$.

False — FSDP is PyTorch's implementation of the same idea as ZeRO-3 (shard params + grads + optimizer states). They are the same strategy under two names.

False — during compute the full layer is temporarily reconstructed via all-gather, so the matrix multiply is full-size. Sharding saves storage, not compute per layer.

False — for a model that already fits, FSDP just adds all-gather + reduce-scatter overhead with no memory benefit to trade against, so it runs ~15–20% slower.

True — both cost about $\frac{N-1}{N}\times \text{size}(x)$ per device; reduce-scatter is essentially all-gather run in reverse with a sum folded in.

True — device $i$ owns shard ${\theta }^{(i)}$ and only its momentum $m^{(i)}$ and variance $v^{(i)}$. That is why the optimizer step is a purely local operation needing no communication.

False — redundancy is eliminated for persistent state, but during a layer's forward/backward every device transiently holds the full layer's parameters at once.

False — FSDP shards params/grads/optimizer; checkpointing attacks activation memory (which FSDP never touches). They are complementary, not substitutes.

The error ignores activation memory, which FSDP does not shard. Activations scale with batch × seq_len × hidden_dim per layer and can dwarf sharded parameter memory — you also need activation checkpointing.

That describes tensor parallelism, not FSDP. FSDP reassembles the full $W$ before the multiply via all-gather; it never leaves $W$ split during compute.

FSDP uses reduce-scatter, not all-reduce. All-reduce would leave every device holding the full gradient (redundant); reduce-scatter sums and hands each device only its own shard's gradient.

The temporary buffer holds one layer's parameters, not the whole model — typically 1–2% of model memory. FSDP frees each layer's gather right after use.

On slow interconnects the all-gather/reduce-scatter time can dominate compute ($T_{\text{comm}}/T_{\text{comp}}$ large), making training slower. Memory savings don't help if the network becomes the bottleneck.

ZeRO-1 shards only optimizer states. Parameter sharding is the ZeRO-3 / FSDP feature; ZeRO-2 adds gradients but still keeps full parameters on each device.

For Adam in fp32, momentum + variance is 16 bytes/param versus 4 for the param — optimizer state is the largest memory consumer, which is exactly why ZeRO starts by sharding it.

Because the full parameters are discarded after the forward pass to reclaim memory; the backward pass needs them again to compute $\partial \mathcal{L}/\partial {\theta }_l$, so it re-fetches them.

Holding every layer's full parameters resident would defeat sharding — memory would climb back to a full replica. The gather-use-discard cycle is the whole point.

Reduce-scatter fuses summation and distribution into one collective, so each device only ever receives its own $\frac{1}{N}$ shard — half the traffic of all-reduce-then-slice and no wasted full-gradient buffer.

Their layers are large matrix multiplies, so compute time is long relative to the fixed communication of gathering that layer — the all-gather overlaps with (and is hidden behind) computation.

Pipelining puts different layer ranges on different device groups, so a layer's parameters only need gathering within its own group, shrinking the collective from all GPUs to intra-stage GPUs.

They cut memory along different axes — mixed precision halves per-tensor byte width, FSDP divides by device count. Multiplying both effects is what makes 175B-scale training feasible.

Accumulation reduces optimizer frequency, but each micro-batch's forward/backward still needs its layers all-gathered, so the gather/scatter traffic per step remains.

Sharding is trivial — the "shard" is the whole tensor and every collective is a no-op. FSDP degenerates to ordinary single-device training with only bookkeeping overhead.

FSDP pads the flattened parameter buffer so it divides evenly; the padding is ignored in compute but means the per-device shard is slightly larger than $\frac{d}{N}$.

Skip FSDP — its all-gather/reduce-scatter overhead has no memory savings to justify it. Reach for FSDP only when model memory exceeds available GPU memory.

The persistent state approaches zero per device, but the un-shardable pieces — the one-layer gather buffer, activations, and communication latency — become the floor, so returns vanish well before that limit.

Barely — with communication fully overlapped by compute the only residual cost is the transient gather buffer, so FSDP behaves almost like DP with a smaller memory footprint.

Add activation checkpointing to recompute activations in backward instead of storing them, trading ~33% more compute for roughly $10\times$ less activation memory — FSDP alone cannot.