NVLink and GPU interconnects

The fundamental problem with PCIe for multi-GPU

WHAT: PCIe connects devices through a hierarchical tree routed through the CPU's PCIe controller. Every GPU-to-GPU transfer must traverse: GPU1 → PCIe → CPU root complex → PCIe → GPU2.

WHY this matters:

• Latency: Multiple hops add microseconds
• Bandwidth: PCIe Gen3 x16 = 15.75 GB/s, Gen4 x16 = 31.5 GB/s
• CPU overhead: The CPU's PCIe lanes are a shared resource
• Scaling: 8 GPUs all trying to synchronize gradients create a traffic jam

The math: If you have 8× V100 GPUs (32GB each) doing distributed training:

• Model size: 1.5 GB of weights
• Per-iteration all-reduce: each GPU must send/receive ~1.5 GB to/from 7 others
• PCIe Gen3: 1.5 GB ÷ 15.75 GB/s ≈ 95 ms per transfer
• With 7 peers: ~665 ms just for communication (vs. ~300 ms compute time)

NVLink architecture derivation

Why differential signaling at high speed?

Start from first principles of electrical signaling:

Single-ended signaling: Voltage relative to ground

• Noise affects signal and ground differently → corrupts measurement
• Ground bounce from switching currents creates common-mode noise
• At multi-GHz frequencies, ground inductance = disaster

Differential signaling: Transmit signal as voltage difference between two wires $V_{\text{signal}} = V_+ - V_-$

WHY this works:

1. Noise couples equally to both wires (common-mode)
2. Receiver subtracts: $(V_+ + n) - (V_- + n) = V_+ - V_-$ (noise cancels!)
3. Can use smaller voltage swings → lower power, higher speed

Bandwidth calculation from physical parameters

Each NVLink consists of multiple differential pairs called lanes. We derive the per-lane payload rate first, then scale up.

Given (NVLink 1.0, Pascal era):

• 8 differential pairs per link direction (8 lanes)
• 20 GT/s (gigatransfers/second) per lane signaling rate
• 8b/10b encoding (8 data bits per 10 transmitted bits)

Derivation: $\text{Payload per lane} = 20 \text{ GT/s} \times \frac{8 \text{ bits}}{10 \text{ bits}} = 16 \text{ Gb/s}$

$\text{Unidirectional link bandwidth} = 8 \text{ lanes} \times 16 \text{ Gb/s} = 128 \text{ Gb/s} = 16 \text{ GB/s}$

NVLink 2.0 / 3.0 (Volta / Ampere era) raise the signaling rate to 25 GT/s:

$\text{Payload per lane} = 25 \text{ GT/s} \times \frac{8 \text{ bits}}{10 \text{ bits}} = 20 \text{ Gb/s}$

$\text{Unidirectional link bandwidth} = 8 \text{ lanes} \times 20 \text{ Gb/s} = 160 \text{ Gb/s} = 20 \text{ GB/s}$

WHY 8b/10b encoding? Need DC balance and clock recovery:

• Can't send long strings of 1s or 0s (no edges → clock drift)
• Encode 8 data bits into 10-bit symbols with guaranteed transitions
• Cost: 20% overhead, but essential for reliable high-speed serial

Bidirectional (full duplex) for NVLink 2.0/3.0: 20 GB/s each direction simultaneously 40 GB/s total per link

For NVLink 2.0, each GPU has 6 NVLink connections, so: $\text{Total GPU NVLink payload} = 6 \times 40 \text{ GB/s} = 240 \text{ GB/s (bidirectional aggregate)}$

Compare to PCIe Gen3 x16 = 16 GB/s → 15× improvement!

Protocol layers: how NVLink actually works

NVLink is not just fast wires—it's a sophisticated protocol stack:

Physical layer (PHY)

• SerDes: Serializer/deserializer converts parallel data to/from serial stream
• Clock recovery: Phase-locked loop (PLL) extracts clock from data transitions
• Equalization: Pre-emphasis and decision feedback equalization (DFE) compensate for cable losses

WHY equalization? At 25 GHz, PCB traces act like low-pass filters: $H(f) \approx e^{-\alpha \sqrt{f} \cdot L}$ where $\alpha$ is loss coefficient, $L$ is length. High frequencies attenuate more → inter-symbol interference.

Pre-emphasis: boost high frequencies at transmitter $V_{\text{tx}}(f) = V_0 \left(1 + \beta \frac{f}{f_0}\right)$

Data link layer (DLL)

• Framing: Packetizes data into flits (flow control units)
• CRC: 16-bit cyclic redundancy check per flit for error detection
• Retry: Automatic retransmission if CRC fails
• Flow control: Credit-based system prevents receiver overflow

Derivation of credit-based flow control:

Let $BDP$ = bandwidth-delay product = amount of data "in flight" $BDP = \text{bandwidth} \times \text{latency}$

For NVLink: 20 GB/s × 500 ns (GPU-to-GPU) = 10 KB

Receiver must have buffer ≥ $BDP$ to keep link saturated. Sender tracks credits:

• Initial credits = buffer size
• Send flit → decrement credit
• Receive ack → increment credit
• If credits = 0 → stall (wait for ack)

Transaction layer

• Load/store semantics: Remote direct memory access (RDMA)
• Atomic operations: Compare-and-swap, fetch-and-add across GPUs
• Coherency protocol: Ensures memory consistency (home-agent snooping)

WHY coherency matters: GPU0 writes X=5 to shared memory GPU1 reads X (expects 5)

Without coherency:

• Write may be buffered in GPU0's cache
• GPU1 reads stale value (X=0)
• Race condition!

NVLink coherency: GPU0 broadcasts write → GPU1 invalidates cache → GPU1 fetches fresh value

NVSwitch: scaling beyond single-server

The problem: With 6 NVLink ports per GPU, you can fully connect at most 6 GPUs directly. What about 16? 128?

The solution: NVSwitch is a crossbar switch with 18 NVLink ports. GPUs connect to the switch, which routes traffic.

Crossbar switch basics

A crossbar switch is a matrix of switches at row-column intersections:

• $N$ inputs, $N$ outputs
• Any input can connect to any output simultaneously (non-blocking)

Bandwidth (using our payload-consistent 20 GB/s/direction per port for NVLink 2.0): $\text{Total switching capacity} = 18 \text{ ports} \times 20 \text{ GB/s} \times 2 \text{ (bidir)} = 720 \text{ GB/s}$

(NVIDIA's headline "900 GB/s" for the NVLink 2.0 NVSwitch uses the 25 GB/s/dir raw line rate: $18 \times 25 \times 2 = 900$. After 8b/10b, usable payload is 720 GB/s.)

Non-blocking property: All 18 ports can transmit/receive at full speed simultaneously (if destinations differ).

Comparing GPU interconnects

Why latency matters more than you think

Intuition: "Bandwidth is all that matters for bulk transfers."

Reality: Small messages dominate in many workloads.

Math: Transfer time = latency + (size / bandwidth)

For 4 KB message:

• PCIe Gen4: 2000 ns + (4 KB ÷ 32 GB/s) = 2000 + 125 = 2125 ns
• NVLink 3.0: 500 ns + (4 KB ÷ 40 GB/s) = 500 + 100 = 600 ns
• Speedup: 3.5× (bandwidth-only ratio predicts just 1.25×!)

For 4 MB message:

• PCIe Gen4: 2000 + 125000 = 127 μs
• NVLink 3.0: 500 + 100000 = 100.5 μs
• Speedup: 1.26× (now bandwidth dominates)

Takeaway: Latency dominates for message sizes < ~1 MB. Most deep learning synchronizations are 100 KB – 10 MB → latency is critical.

Recall Explain to a curious 12-year-old

Imagine you have 8 super-fast workers (GPUs) who need to team up to solve a giant puzzle (train an AI). Each worker is incredibly quick on their own piece, but they need to share information constantly: "Hey, I found this edge piece!" "My section looks like sky!"

The problem: If they have to shout across a big warehouse through one tiny walkie-talkie (PCIe bus), only one can talk at a time, and it takes forever. By the time everyone has shared their updates, they could have already done way more puzzle work!

NVLink's solution: Give each worker direct phone lines to all the other workers (point-to-point connections). Now they can all talk simultaneously, share big chunks of information super fast, and spend most of their time actually solving the puzzle instead of waiting to communicate.

It's like the difference between:

• Old way: Playing a multiplayer game on dial-up internet (slow, laggy)
• NVLink: Playing on fiber optic internet (fast, smooth)

The puzzle gets solved WAY faster because the workers spend less time coordinating and more time working!

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho beta, yahaan core baat ye hai ki jab hum multiple GPUs ko ek saath use karte hain—jaise deep learning training mein—tab GPUs ko aapas mein bahut bada data share karna padta hai, jaise gradients aur weights. Problem ye hai ki traditional PCIe bus bahut slow hai iske liye. Socho GPU ki apni internal memory bandwidth 900+ GB/s hai, lekin PCIe Gen3 sirf ~16 GB/s deta hai. Matlab jaise samundar ka paani ek patli si garden hose se nikalne ki koshish kar rahe ho! Upar se PCIe mein har GPU-to-GPU transfer CPU ke through jaata hai, toh extra hops aur latency add ho jaati hai. Isi bottleneck ko solve karne ke liye NVIDIA ne banaya NVLink—ek direct, high-speed connection jo GPUs ko aapas mein CPU ko bypass karke baat karne deta hai.

Ab thodi technical intuition samajh lo. NVLink ke andar differential pairs (lanes) hoti hain jo signals bhejti hain. Differential signaling ka matlab hai ki data do wires ke voltage difference (V+ minus V-) ke roop mein bheja jaata hai, single wire pe nahi. Iska fayda ye ki agar koi noise aata hai, toh wo dono wires pe barabar aata hai, aur receiver subtract karke usse cancel kar deta hai—isliye high speed pe bhi signal clean rehta hai. Phir 8b/10b encoding use hoti hai jisme 8 data bits ko 10 bits mein bhej te hain taaki clock recovery aur DC balance maintain rahe (warna lambi 1s ya 0s ki string se clock drift ho jaata). Bas isme 20% overhead lagta hai, isiliye "25 GB/s" wala marketing number actually 20 GB/s usable payload deta hai—ye chhota sa point exams aur real understanding dono mein kaam aata hai.

Ye topic important kyun hai? Kyunki aaj ke AI aur HPC (high performance computing) systems poori tarah multi-GPU setups pe chalte hain, aur inki asli speed sirf compute power se nahi, balki GPUs ke beech ki communication speed se decide hoti hai. Agar communication slow hoga toh mehnga GPU compute bekaar baitha rahega intezaar karte hue. Toh NVLink jaise interconnects samajhna tumhe ye clarity dega ki modern data centers aur AI training clusters kaise itni fast scale karte hain—ye ek foundational concept hai jo hardware aur ML dono fields mein tumhare kaam aayega.