NVLink and GPU interconnects — Hinglish notes

6.3.4 · Hardware › Interconnects, Buses & SoC

PCIe ka multi-GPU ke liye fundamental problem

KYA HAI: PCIe devices ko CPU ke PCIe controller se guzarte ek hierarchical tree ke through connect karta hai. Har GPU-to-GPU transfer yeh path leti hai: GPU1 → PCIe → CPU root complex → PCIe → GPU2.

YEH KYUN MATTER KARTA HAI:

Latency: Multiple hops microseconds add karte hain
Bandwidth: PCIe Gen3 x16 = 15.75 GB/s, Gen4 x16 = 31.5 GB/s
CPU overhead: CPU ke PCIe lanes ek shared resource hain
Scaling: 8 GPUs sab ek saath gradients synchronize karne ki koshish karte hain to traffic jam ho jaata hai

Math: Agar tumhare paas 8× V100 GPUs (32GB each) hain jo distributed training kar rahe hain:

Model size: 1.5 GB weights
Per-iteration all-reduce: har GPU ko ~1.5 GB send/receive karna hoga 7 doosron se
PCIe Gen3: 1.5 GB ÷ 15.75 GB/s ≈ 95 ms per transfer
7 peers ke saath: ~665 ms sirf communication ke liye (vs. ~300 ms compute time)

NVLink architecture derivation

High speed par differential signaling kyun?

Electrical signaling ke first principles se shuru karte hain:

Single-ended signaling: Ground ke relative voltage

Noise signal aur ground ko alag-alag affect karta hai → measurement corrupt ho jaata hai
Switching currents se ground bounce common-mode noise create karta hai
Multi-GHz frequencies par, ground inductance = disaster

Differential signaling: Signal ko do wires ke beech voltage difference ke roop mein transmit karo $V_{signal} = V_{+} - V_{-}$

YEH KYUN KAAM KARTA HAI:

Noise dono wires par equally couple hota hai (common-mode)
Receiver subtract karta hai: $(V_{+} + n) - (V_{-} + n) = V_{+} - V_{-}$ (noise cancel ho jaata hai!)
Chhote voltage swings use kar sakte hain → kam power, zyada speed

Physical parameters se bandwidth calculation

Har NVLink multiple differential pairs se bana hota hai jinhein lanes kehte hain. Pehle per-lane payload rate derive karte hain, phir scale up karte hain.

Given (NVLink 1.0, Pascal era):

8 differential pairs per link direction (8 lanes)
20 GT/s (gigatransfers/second) per lane signaling rate
8b/10b encoding (10 transmitted bits mein 8 data bits)

Derivation: $Payload per lane = 20 GT/s \times \frac{8 bits}{10 bits} = 16 Gb/s$

$Unidirectional link bandwidth = 8 lanes \times 16 Gb/s = 128 Gb/s = 16 GB/s$

NVLink 2.0 / 3.0 (Volta / Ampere era) signaling rate ko 25 GT/s tak raise karta hai:

$Payload per lane = 25 GT/s \times \frac{8 bits}{10 bits} = 20 Gb/s$

$Unidirectional link bandwidth = 8 lanes \times 20 Gb/s = 160 Gb/s = 20 GB/s$

8b/10b encoding kyun? DC balance aur clock recovery ke liye zaroori hai:

Lambi strings of 1s ya 0s nahi bhej sakte (edges nahi → clock drift)
8 data bits ko 10-bit symbols mein encode karo guaranteed transitions ke saath
Cost: 20% overhead, lekin reliable high-speed serial ke liye zaroori

Bidirectional (full duplex) NVLink 2.0/3.0 ke liye: 20 GB/s har direction mein simultaneously 40 GB/s total per link

NVLink 2.0 ke liye, har GPU mein 6 NVLink connections hote hain, isliye: $Total GPU NVLink payload = 6 \times 40 GB/s = 240 GB/s (bidirectional aggregate)$

PCIe Gen3 x16 = 16 GB/s se compare karo → 15× improvement!

Worked example DGX-1 mein NVLink 2.0 topology

Setup: 8× Tesla V100 GPUs, har ek mein 6 NVLink ports Topology: Hybrid cube-mesh

Har GPU 4 neighbours se ek ring mein connect hota hai
2 additional cross-links shortcuts create karte hain Yeh topology kyun?

Ring: Guarantee karta hai ki har GPU har doosre tak pahunch sake (max 4 hops)
Cross-links: Average hop count 2.7 se 1.9 tak reduce karte hain
No single point of failure: Kisi bhi pair ke beech multiple paths Bandwidth analysis: GPU0 → GPU4

Direct path: 1 hop × 20 GB/s = 20 GB/s effective payload
PCIe se: ~8 GB/s (CPU bottleneck)
Speedup: 2.5× is pair ke liye

All-reduce operation (8 GPUs mein gradients sum karo):

Optimal ring algorithm: 7 steps
Har step mein: 1.5 GB transfer sab links par parallel mein
Time: 7 × (1.5 GB ÷ 20 GB/s) = 525 ms
PCIe equivalent (~15 GB/s lekin CPU-serialized, effective ~5 GB/s): ~5250 ms
Speedup: ~10×

Formula NVLink bandwidth vs. generation (usable payload)

Generation	Signaling Rate	Encoding	Lanes	Payload / dir	Bidir / link	Year
NVLink 1.0	20 GT/s	8b/10b	8	16 GB/s	32 GB/s	2016
NVLink 2.0	25 GT/s	8b/10b	8	20 GB/s	40 GB/s	2017
NVLink 3.0	25 GT/s	8b/10b	8	20 GB/s	40 GB/s	2020
NVLink 4.0	26.56 GT/s (PAM4, 2b/symbol)	128b/130b	9	~46.5 GB/s	~93 GB/s	2022

"50 GB/s" datasheets par note: NVIDIA NVLink 2.0/3.0 ke liye 50 GB/s bidirectional per link advertise karta hai raw 25 GT/s line rate use karke (25 GB/s/dir × 2). 8b/10b ke baad yeh 40 GB/s bidirectional payload ban jaata hai. Hum consistency ke liye payload numbers use karte hain.

Key insight: NVLink 4.0 (H100) ne PAM4 signaling aur 128b/130b encoding ki taraf switch kiya. $NVLink 4.0 payload/dir = 9 lanes \times 26.56 GT/s \times 2 \frac{bits}{symbol} \times \frac{128}{130} \div 8 \approx 46.5 GB/s$

PAM4 kyun? 4 voltage levels per symbol → 2 bits/symbol → faster switching ke bina data rate double ho jaati hai.

Protocol layers: NVLink actually kaise kaam karta hai

NVLink sirf fast wires nahi hai — yeh ek sophisticated protocol stack hai:

Physical layer (PHY)

SerDes: Serializer/deserializer parallel data ko serial stream mein aur serial se parallel mein convert karta hai
Clock recovery: Phase-locked loop (PLL) data transitions se clock extract karta hai
Equalization: Pre-emphasis aur decision feedback equalization (DFE) cable losses ko compensate karte hain

Equalization kyun? 25 GHz par, PCB traces low-pass filters ki tarah act karte hain: $H (f) \approx e^{- α f \cdot L}$ jahan $α$ loss coefficient hai, $L$ length hai. High frequencies zyada attenuate hoti hain → inter-symbol interference.

Pre-emphasis: transmitter par high frequencies boost karo $V_{tx} (f) = V_{0} (1 + β \frac{f}{f _{0}})$

Data link layer (DLL)

Framing: Data ko flits (flow control units) mein packetize karta hai
CRC: Error detection ke liye har flit par 16-bit cyclic redundancy check
Retry: Agar CRC fail ho to automatic retransmission
Flow control: Credit-based system receiver overflow prevent karta hai

Credit-based flow control ka derivation:

Maano $B D P$ = bandwidth-delay product = "in flight" data ki matra $B D P = bandwidth \times latency$

NVLink ke liye: 20 GB/s × 500 ns (GPU-to-GPU) = 10 KB

Receiver ka buffer ≥ $B D P$ hona chahiye link ko saturated rakhne ke liye. Sender credits track karta hai:

Initial credits = buffer size
Flit bhejna → credit decrement karo
Ack receive karna → credit increment karo
Agar credits = 0 → stall (ack ka wait karo)

Transaction layer

Load/store semantics: Remote direct memory access (RDMA)
Atomic operations: GPUs ke across compare-and-swap, fetch-and-add
Coherency protocol: Memory consistency ensure karta hai (home-agent snooping)

Coherency kyun matter karta hai: GPU0 shared memory mein X=5 likhta hai GPU1 X read karta hai (5 expect karta hai)

Coherency ke bina:

Write GPU0 ke cache mein buffered reh sakta hai
GPU1 stale value padhta hai (X=0)
Race condition!

NVLink coherency: GPU0 write broadcast karta hai → GPU1 cache invalidate karta hai → GPU1 fresh value fetch karta hai

Common mistake Yeh assume karna ki NVLink automatically sab kuch faster bana deta hai

Galat intuition: "Main GPUs ko NVLink se connect kar dunga aur mera code 10× faster ho jaayega."

Yeh sahi kyun lagta hai: Marketing materials mein huge speedups dikhte hain!

Asli baat: NVLink tabhi help karta hai jab tumhara code communication-bound ho, compute-bound nahi.

Galti ko steel-man karna: Agar tumhare kernels 100ms chalte hain aur sirf har 100ms mein synchronize karte hain (1MB transfer karke), tab:

PCIe time: 1 MB ÷ 15 GB/s = 0.067 ms
NVLink time: 1 MB ÷ 20 GB/s = 0.05 ms
Speedup: 0.017 ms bacha 100ms mein se = 0.017% improvement

NVLink kab shine karta hai:

Chhoti frequent synchronizations (har kuch ms mein gradient updates)
Large model-parallel models (GPUs alag-alag layers hold karte hain)
Sparse all-to-all communication patterns Fix: Pehle profile karo. Actual NVLink utilization measure karne ke liye nvidia-smi nvlink -i 0 -s d use karo. Agar <20% hai, tum compute-bound ho → NVLink help nahi karega.

NVSwitch: single-server se aage scaling

Problem: Har GPU mein 6 NVLink ports ke saath, seedha zyada se zyada 6 GPUs fully connect kar sakte ho. 16 ke baare mein? 128?

Solution: NVSwitch ek crossbar switch hai jisme 18 NVLink ports hain. GPUs switch se connect hote hain, jo traffic route karta hai.

Crossbar switch basics

Ek crossbar switch row-column intersections par switches ka matrix hota hai:

$N$ inputs, $N$ outputs
Koi bhi input kisi bhi output se simultaneously connect ho sakta hai (non-blocking)

Bandwidth (NVLink 2.0 ke liye hamare payload-consistent 20 GB/s/direction per port use karte hue): $Total switching capacity = 18 ports \times 20 GB/s \times 2 (bidir) = 720 GB/s$

(NVIDIA ka headline "900 GB/s" NVLink 2.0 NVSwitch ke liye 25 GB/s/dir raw line rate use karta hai: $18 \times 25 \times 2 = 900$ . 8b/10b ke baad, usable payload 720 GB/s hai.)

Non-blocking property: Sab 18 ports simultaneously full speed par transmit/receive kar sakte hain (agar destinations alag hon).

GPU interconnects ka comparison

Formula Interconnect comparison table (bidirectional payload)

Technology	Bandwidth (bidir)	Latency	Topology	Use case
PCIe Gen3 x16	32 GB/s	~2 μs	Tree	Standard device I/O
PCIe Gen4 x16	64 GB/s	~2 μs	Tree	Current standard
PCIe Gen5 x16	128 GB/s	~2 μs	Tree	Emerging (2023+)
NVLink 2.0 (per link)	40 GB/s	~500 ns	Mesh	Multi-GPU (V100)
NVLink 3.0 (per link)	40 GB/s	~500 ns	Mesh	Multi-GPU (A100)
NVLink 4.0 (per link)	~93 GB/s	~500 ns	Mesh	Multi-GPU (H100)
AMD Infinity Fabric	~92 GB/s	~600 ns	Mesh	Multi-GPU (MI250X)
Intel Xe Link	~128 GB/s	~1 μs	Mesh	Multi-GPU (Ponte Vecchio)

Note: PCIe rows line-rate hain (128b/130b overhead negligible ~1.5% hai); NVLink 2.0/3.0 rows 8b/10b-adjusted payload dikhate hain. Datasheets NVLink 2.0/3.0 ke liye 50 GB/s bidir (raw) quote karte hain.

Key insight: GPU interconnects complexity/cost ka trade-off karte hain PCIe se 4-10× better bandwidth + lower latency ke liye.

Latency tumhare sochne se zyada kyun matter karta hai

Intuition: "Bulk transfers ke liye bandwidth hi sab kuch hai."

Asli baat: Bahut saare workloads mein chhote messages dominate karte hain.

Math: Transfer time = latency + (size / bandwidth)

4 KB message ke liye:

PCIe Gen4: 2000 ns + (4 KB ÷ 32 GB/s) = 2000 + 125 = 2125 ns
NVLink 3.0: 500 ns + (4 KB ÷ 40 GB/s) = 500 + 100 = 600 ns
Speedup: 3.5× (sirf bandwidth ratio se toh sirf 1.25× predict hota!)

4 MB message ke liye:

PCIe Gen4: 2000 + 125000 = 127 μs
NVLink 3.0: 500 + 100000 = 100.5 μs
Speedup: 1.26× (ab bandwidth dominate karta hai)

Takeaway: ~1 MB se chhote message sizes ke liye latency dominate karta hai. Zyaadatar deep learning synchronizations 100 KB – 10 MB hoti hain → latency critical hai.

Recall Ek curious 12-saal ke bacche ko explain karo

Socho tumhare paas 8 super-fast workers (GPUs) hain jo ek giant puzzle (AI train karna) solve karne ke liye team up karte hain. Har worker apne piece par incredibly quick hai, lekin unhe lagaataar information share karni padti hai: "Hey, maine yeh edge piece dhundha!" "Mera section sky jaisa lagta hai!"

Problem: Agar unhe ek bade warehouse ke across ek tiny walkie-talkie (PCIe bus) se shout karna pade, toh ek waqt mein sirf ek hi bol sakta hai, aur bahut time lag jaata hai. Jab tak sabne apne updates share kiye hote hain, woh already bahut zyada puzzle work kar sakte the!

NVLink ka solution: Har worker ko baaki sab workers ki direct phone lines do (point-to-point connections). Ab woh sab simultaneously baat kar sakte hain, information ke bade chunks super fast share kar sakte hain, aur zyaadatar time actually puzzle solve karne mein laga sakte hain communicate karne ka wait karne ki jagah.

Yeh farq hai:

Purana tarika: Multiplayer game dial-up internet par khelna (slow, laggy)
NVLink: Fiber optic internet par khelna (fast, smooth)

Puzzle WAY faster solve hoti hai kyunki workers coordinating mein kam time aur kaam karne mein zyada time lagate hain!