FSDP and sharded training

6.1.9 · AI-ML › Scaling & Efficient Architectures

FSDP Actually Kya Karta Hai

Fully Sharded Data Parallel ek memory optimization strategy hai jo teen key components ko shard (partition) karta hai data-parallel workers mein:

1. Model parameters ($\theta$)
2. Gradients (${\nabla }_{\theta }\mathcal{L}$)
3. Optimizer states (momentum, variance for Adam)

Traditional data parallelism har device par teeno ki full copies rakhta hai. FSDP sirf $\frac{1}{N}$ rakhta hai har ek ka, $N$ devices mein se har ek par.

FSDP Kaise Kaam Karta Hai: Teen-Phase Protocol

Phase 1: Forward Pass

Har layer ki computation se pehle:

1. Device $i$ ke paas shard ${\theta }_l^{(i)}$ hai layer $l$ ke liye
2. All-gather operation: saare devices se $\{{\theta }_l^{(0)},{\theta }_l^{(1)},\dots ,{\theta }_l^{(N-1)}\}$ collect karo
3. Device memory mein temporarily poora ${\theta }_l$ reconstruct karo
4. Layer ka forward pass compute karo: $h_{l+1}=f_l(h_l,{\theta }_l)$
5. Borrowed shards discard karo, sirf ${\theta }_l^{(i)}$ rakho

Yeh step kyun? Har layer ko apne saare parameters chahiye hote hain activations sahi se compute karne ke liye. Hum matrix multiply $y=Wx$ split nahi kar sakte agar hamare paas sirf aadha $W$ ho.

Phase 2: Backward Pass

Har layer ke liye reverse mein:

1. ${\theta }_l$ ko dobara all-gather karo (gradient computation ke liye zaroori)
2. Local gradients compute karo: $\frac{\partial \mathcal{L}}{\partial {\theta }_l}$
3. Gradients ko reduce-scatter karo: har device ko sirf ${\nabla }_{{\theta }_l^{(i)}}$ milta hai (apne shard ka gradient)
4. Poora ${\theta }_l$ phir se discard karo

Reduce-scatter kyun? Poora gradient compute karne ke baad, hum devices ke contributions ko sum karte hain (reduce) aur chunks distribute karte hain (scatter), taaki har device sirf apna shard update kare.

Phase 3: Optimizer Step

Har device independently apna shard update karta hai: ${\theta }_l^{(i)}\leftarrow {\theta }_l^{(i)}-\eta \cdot \text{Adam}({\nabla }_{{\theta }_l^{(i)}},m_l^{(i)},v_l^{(i)})$

Optimizer states $(m,v)$ bhi sharded hain, isliye har device sirf $\frac{1}{N}$ store karta hai momentum/variance buffers ka.

Communication Primitives

FSDP do collective operations par rely karta hai:

ZeRO ke Saath Comparison

FSDP, PyTorch ka implementation hai ZeRO (Zero Redundancy Optimizer) ka, jo Microsoft DeepSpeed se aaya hai:

• ZeRO Stage 1: Sirf optimizer states shard karo → $4\times$ memory reduction
• ZeRO Stage 2: Optimizer states + gradients shard karo → $8\times$ memory reduction
• ZeRO Stage 3: Optimizer states + gradients + parameters shard karo → $N\times$ memory reduction (FSDP = ZeRO-3)

Teen stages kyun? Har stage zyada communication overhead introduce karta hai. ZeRO-1 moderate models ke liye faster hai; ZeRO-3 sirf tab zaroori hai jab models single-GPU memory se bade hon.

Hybrid Sharding Strategies

Modern frameworks hybrid parallelism allow karte hain:

FSDP + Pipeline Parallelism:

• Model ko stages mein partition karo (layers 1-12 GPU group A par, layers 13-24 GPU group B par)
• Har stage ke andar FSDP se shard karo
• Benefit: All-gather scope ko intra-stage devices tak reduce karo

FSDP + Tensor Parallelism:

• Bade layers (jaise 12,288 × 12,288 weight matrix) ko ek node ke andar GPUs mein split karo
• Nodes ke across FSDP se parameters shard karo
• Benefit: Computation ke liye tensor parallelism, memory ke liye FSDP

Effective parallelism ka formula: $\text{Total GPUs}=N_{\text{data}}\times N_{\text{pipeline}}\times N_{\text{tensor}}$

Implementation Details

Wrapping granularity: FSDP individual layers, modules, ya poore model ko wrap kar sakta hai. Finer wrapping temporary memory kam karta hai (chhote all-gather buffers) lekin communication frequency badh jaati hai.

Typical strategy:

model = TransformerModel(...)
for layer in model.transformer_layers:
    layer = FSDP(layer)  # Wrap each transformer block
model = FSDP(model)  # Outer wrap for top-level

Mixed precision: FSDP shards fp32 mein store karta hai (optimizer ke liye) lekin fp16 mein communicate karta hai (bandwidth ke liye). All-gather, send karne se pehle fp32 → fp16 convert karta hai.

CPU offloading: Kuch FSDP implementations inactive shards ko CPU RAM mein offload karne ki allow karte hain, GPU memory ke liye PCIe bandwidth trade karte hue. Severe latency penalties ki wajah se rarely use hota hai.

Connections

• Data Paralelism — FSDP, data parallelism ko sharding ke saath extend karta hai
• Model Parallelism — FSDP orthogonal hai; dono combine kar sakte hain
• Mixed Precision Training — FSDP fp16 communication, fp32 computation leverage karta hai
• Gradient Accumulation — Per-step communication kam karne ke liye microbatches par accumulate karo
• Activation Checkpointing — Activation memory ke liye FSDP ka essential complement
• ZeRO Optimizer — FSDP, DeepSpeed ka ZeRO-3 implement karta hai
• Communication Collectives — All-gather, reduce-scatter FSDP ke primitives hain
• Pipeline Parallelism — Hybrid: pipeline + FSDP within stages

#flashcards/ai-ml

What are the three components FSDP shards across devices? :: Model parameters, gradients, aur optimizer states (momentum, variance).

Concept Map