Worked examples — NVLink and GPU interconnects

This page is a drill hall. The parent note built the theory; here we hammer every kind of question NVLink arithmetic can throw at you, one worked example per case. Before each solution I ask you to Forecast the answer — guessing first is how the number sticks.

The scenario matrix

Cell	Case class	What makes it tricky	Example
A	8b/10b payload from line rate	must multiply by $8/10$	Ex 1
B	PAM4 + 128b/130b payload	2 bits/symbol AND new encoding ratio	Ex 2
C	Raw vs payload (datasheet twist)	headline number ≠ usable number	Ex 3
D	Direction bookkeeping (uni vs bidir)	factor-of-2 trap	Ex 4
E	Aggregate over many links	multiply per-link by link count	Ex 5
F	Saturated link (buffer ≥ BDP)	efficiency = 100%	Ex 6
G	Degenerate: buffer too small (< BDP)	stall, efficiency < 100%	Ex 7
H	Limiting case: zero latency / infinite buffer	boundary sanity check	Ex 8
I	Real-world all-reduce word problem	ring algorithm, many peers	Ex 9
J	Exam twist: NVLink vs PCIe crossover	when is NVLink NOT worth it?	Ex 10

Every cell below is filled. Let us begin.

---

Building the one tool we reuse everywhere

Almost every example uses the same conversion chain, so define it once, in plain words, before any symbol appears.

Read the figure below as a pipe that narrows at each stage. Start at the wide mouth on the left — the raw $8\times 20=160$ GT/s of symbols. The pipe keeps its width through the "×1 bit/symbol" stage (on/off signaling adds nothing). At the orange "×8/10" stage the pipe narrows: one fifth of the width is lost to encoding overhead, dropping us to 128 Gb/s. Finally the green "÷8" stage narrows it again to convert bits into bytes, ending at the 16 GB/s spout. The visual point: each stage can only shrink the flow, never grow it — so your final payload is always the smallest number in the chain, and forgetting a stage means over-reporting.

---

Cell A — 8b/10b payload

1. Bits per lane per second $=20\text{GT/s}\times 1\frac{\text{bit}}{\text{symbol}}\times \frac{8}{10}=16\text{Gb/s}$. Why this step? Turns raw symbols into real payload bits by paying the encoding tax first.
2. All 8 lanes $=8\times 16=128\text{Gb/s}$. Why this step? Lanes run in parallel, so their payloads simply add.
3. Gigabits → gigabytes: $128÷8=16\text{GB/s}$. Why this step? A byte is 8 bits; datasheets speak GB/s.

Verify: Plug into master formula: $8\times 20\times 1\times \frac{8}{10}\times \frac{1}{8}=16$. ✓ Units: $\text{lanes}\cdot \frac{\text{Gsym}}{s}\cdot \frac{\text{bit}}{\text{sym}}\cdot \frac{\text{byte}}{8\text{bit}}=\text{GB/s}$. ✓ Matches parent table row NVLink 1.0.

---

Cell B — PAM4 + 128b/130b

Before the example, here are the raw parent-note figures reproduced so you need not context-switch:

1. Bits/lane/s $=26.56\times 2\times \frac{128}{130}=52.30\text{Gb/s}$. Why this step? Two things changed at once — bits/symbol AND the encoding ratio — so both enter here.
2. 9 lanes: $9\times 52.30=470.7\text{Gb/s}$. Why this step? Parallel lanes add; note the odd lane count (9, not 8).
3. Gigabits → gigabytes: $470.7÷8=58.8\text{GB/s}$. Why this step? Exactly as in Ex 1 — a byte is 8 bits, so we divide the bit-rate by 8 to report bytes. The result is in GB/s, not Gb/s.
4. Reconcile with the parent's 46.5: the parent's datasheet-consistent figure is $\approx 46.5$ GB/s/dir. The pure master-formula number is 58.8 GB/s; real silicon includes extra framing/protocol overhead the marketing payload folds in. For this drill we report the formula value 58.8 GB/s and flag that datasheet-consistent NVLink 4.0 is quoted near 46.5.

Verify: $9\times 26.56\times 2\times \frac{128}{130}÷8=58.8$ GB/s (formula). ✓ The $÷8$ and $b=2$ bits/symbol are the only structural differences from Ex 1. The gap to 46.5 is protocol overhead beyond encoding — see the mistake box.

---

Cell C — raw line rate vs payload (the datasheet twist)

1. Decode the 25: $25\text{GT/s}\times 1\frac{\text{bit}}{\text{sym}}÷8=3.125\text{GB/s}$ per lane of raw symbols. Over 8 lanes: $8\times 3.125=25\text{GB/s}$ raw. Why this step? Shows "25" is symbols-as-bytes, before the encoding haircut.
2. Apply 8b/10b: $25\times \frac{8}{10}=20\text{GB/s}$ payload. Why this step? Only $8/10$ of transmitted bits are your data.

Verify: $8\times 25\times 1\times \frac{8}{10}÷8=20$ GB/s. ✓ The 25 is the pre-encoding line rate; the 20 is what your data actually moves at. Always convert before timing calculations.

---

Cell D — direction bookkeeping

1. Full duplex means simultaneous: send-lanes and receive-lanes are physically separate, so directions do not share bandwidth. Why this step? This is why you may add them; on a half-duplex bus you could not.
2. Add: $20+20=40\text{GB/s}$ bidirectional.

Verify: $2\times 20=40$ GB/s. ✓ Matches parent table "Bidir/link = 40 GB/s" for NVLink 2.0. Trap: never use 40 GB/s to time a one-way transfer — a single flow only ever sees 20.

---

Cell E — aggregate over many links

1. Per-link bidir = 40 GB/s (from Ex 4).
2. Six links = $6\times 40=240\text{GB/s}$ aggregate. Why this step? Links are independent physical ports; their bandwidths add.

Verify: $6\times 40=240$ GB/s. ✓ Parent says 240 GB/s aggregate, and $240/16=15\times$ over PCIe Gen3 x16. ✓ Caution: aggregate ≠ what any single peer sees; one peer is capped at 40 (or 20 one-way).

---

Cell F — saturated link (efficiency 100%)

First define the two tools that govern saturation: the BDP and the efficiency law we will lean on in Cells F, G and H.

1. Compute BDP $=20\frac{\text{GB}}{\text{s}}\times 500\text{ns}=20\times 10^9\times 500\times 10^{-9}=10,000\text{B}=10\text{KB}$ (decimal KB). Why this step? BDP is the minimum buffer for zero stalls — the yardstick.
2. Apply the efficiency law: $\min \left(1,\frac{16}{10}\right)=\min (1,1.6)=1=100\%$. Why this step? Buffer ≥ BDP means the ratio ≥ 1, so the cap fires and there is no stall.
3. Cross-check with a timeline: buffer drains in $16\text{KB}÷20\text{GB/s}=800\text{ns}$; the first ack returns at $500\text{ns}<800\text{ns}$, replenishing credits before exhaustion. Why this step? Confirms the efficiency-law verdict against the actual clock.

Verify: ack at 500 ns arrives before buffer empties at 800 ns → no stall. Efficiency $=\min (1,16/10)=100\%$. ✓

Read the figure below as the sender's fill-and-refill timeline. The blue curve is the amount of data "in flight" as time advances (rising because the sender is pouring at 20 GB/s). The gray dashed line at 10 KB marks the BDP; the orange dashed line at 16 KB marks the buffer ceiling. The green dotted vertical at 500 ns is when the first acknowledgement returns. The key thing to see: the green line arrives to the left of the point where the blue curve would hit the 16 KB ceiling (at 800 ns), so credits are refreshed before the buffer ever empties — the blue curve never flattens, meaning the link never idles. That geometric "ack-before-ceiling" is exactly the condition buffer ≥ BDP.

---

Cell G — degenerate: buffer too small (stall)

1. Time to send 6 KB $=6\times 10^3\text{B}÷20\times 10^9\text{B/s}=300\text{ns}$. Why this step? When credits hit zero, this is how long useful work lasted.
2. Ack arrives at 500 ns; sender idle from 300 → 500 ns. Why this step? The gap is the forced stall.
3. Stall $=500-300=200\text{ns}$.
4. Efficiency $=\min \left(1,\frac{6}{10}\right)=0.60=60\%$, matching $\frac{\text{busy}}{\text{cycle}}=\frac{300}{500}$. Why this step? Below BDP the cap does not fire, so efficiency is just the raw ratio buffer/BDP.

Verify: $300/500=0.6$ and $6/10=0.6$ agree. ✓ Effective throughput $=0.6\times 20=12$ GB/s. ✓ Rule confirmed: below BDP, efficiency $=\text{buffer}/\text{BDP}=60\%$. ✓

---

Cell H — limiting cases (boundary sanity)

1. (a) $\text{BDP}=20\text{GB/s}\times 0\text{ns}=0\text{B}$. Then $\text{efficiency}=\min (1,B/0)=\min (1,\infty )=1=100\%$ for any $B>0$. Why this step? Tests the law at the low boundary — no in-flight data means the buffer requirement collapses to zero.
2. (b) As $B\to \infty$, $B/\text{BDP}\to \infty$, so $\text{efficiency}=\min (1,\infty )=1=100\%$. Why this step? Shows the $\min (1,\cdot )$ cap keeps efficiency at exactly 100%, never above it — a sanity guard.

Verify: $\min (1,\infty )=1$ in both limits. ✓ Both degenerate ends give 100%, matching intuition: only a finite latency paired with a too-small buffer (Cell G) can stall.

---

Cell I — real-world all-reduce word problem

Before the numbers, fix the algorithm carefully — the common trap is assuming each hop moves the whole gradient.

1. Steps $=2(N-1)=2\times (8-1)=14$ (reduce-scatter + all-gather). Why this step? The ring completes a reduction in two $N-1$-step passes, not one; both must be counted.
2. Chunk per step $=M/N=1.5\text{GB}/8=0.1875\text{GB}$. Why this step? This is the whole correction — a hop carries one chunk, not the full gradient.
3. Per-step time $=0.1875\text{GB}÷20\text{GB/s}=9.375\times 10^{-3}\text{s}=9.375\text{ms}$. Why this step? All links move their chunk in parallel, so one step costs a single chunk-transfer time.
4. Total NVLink $=14\times 9.375\text{ms}=131.25\text{ms}$. Why this step? Steps run one after another; multiply.
5. PCIe (5 GB/s effective): per step $=0.1875/5=37.5\text{ms}$; total $=14\times 37.5=525\text{ms}$. Why this step? Same algorithm and data volume, slower link.

Verify: NVLink $=2(8-1)\times \frac{1.5/8}{20}=14\times 9.375\text{ms}=131.25$ ms. ✓ PCIe $=14\times 37.5=525$ ms. ✓ Speedup $=525/131.25=4.0\times$. Note this corrects a naive model (full 1.5 GB per hop) which would have over-estimated NVLink time by a factor of $\frac{N}{2(N-1)/N}\approx$ several ×; the parent's ~10× headline uses a more pessimistic PCIe figure and the "full-gradient" simplification, so its arithmetic and ours differ — the method here is the correct one. Units: GB ÷ (GB/s) = s throughout. ✓

---

Cell J — exam twist: NVLink vs PCIe crossover

1. Model each path as fixed cost + transfer time: NVLink time $=C+S/20$; PCIe time $=S/5$. Why this step? NVLink's advantage is speed but it pays a setup tax; PCIe's fixed cost is ~0 here.
2. Set equal: $C+\frac{S}{20}=\frac{S}{5}$. Why this step? The crossover is exactly where the two times match.
3. Solve: $C=\frac{S}{5}-\frac{S}{20}=\frac{4S-S}{20}=\frac{3S}{20}$, so $S=\frac{20C}{3}$.
4. Plug $C=50\mu \text{s}=50\times 10^{-6}\text{s}$: $S=\frac{20\times 50\times 10^{-6}}{3}\text{GB}=3.33\times 10^{-4}\text{GB}=333\text{KB}$ (decimal KB). Why this step? Below 333 KB the setup overhead makes NVLink slower than plain PCIe.

Verify: At $S=333\text{KB}=3.33\times 10^{-4}$ GB: NVLink $=50\mu s+3.33\times 10^{-4}/20\text{s}=50+16.67=66.67\mu s$; PCIe $=3.33\times 10^{-4}/5=66.67\mu s$. Equal ✓.

Read the figure below as two cost lines crossing. The blue line (NVLink) starts above zero at $S=0$ — that raised intercept is the fixed 50 µs setup tax — then climbs gently (shallow slope $1/20$). The orange line (PCIe) starts at the origin but climbs steeply (slope $1/5$, four times steeper). They meet at the red dot: $S=333$ KB, $66.7$ µs. To the left of the dot the orange line is lower — PCIe wins for small messages. To the right the blue line is lower — NVLink wins for large messages. The lesson lives in that intersection: NVLink shines for large transfers (gradients, activations); for chatty tiny messages the setup tax can flip the winner.

---

Closing the drill hall — lessons from the ten cells

Recall Self-test: name the cell, then answer

NVLink 1.0 8-lane 20 GT/s 8b/10b unidirectional payload? ::: 16 GB/s (Cell A) NVLink 4.0 raw master-formula payload/dir (9 lanes, 26.56 GT/s, PAM4, 128b/130b)? ::: 58.8 GB/s formula (46.5 datasheet) (Cell B) A datasheet "25 GB/s/dir" for NVLink 2.0 corresponds to what payload? ::: 20 GB/s after 8b/10b (Cell C) One V100's aggregate NVLink 2.0 bandwidth (6 links)? ::: 240 GB/s bidirectional (Cell E) BDP for 20 GB/s at 500 ns latency? ::: 10 KB decimal (Cell F) Efficiency with a 6 KB buffer when BDP is 10 KB? ::: 60% via min(1, 6/10) (Cell G) Ring all-reduce time, 8 GPUs, 1.5 GB, 20 GB/s (chunks, both phases)? ::: 131.25 ms (Cell I) Crossover payload where NVLink overhead 50 µs equals PCIe (5 vs 20 GB/s)? ::: 333 KB (Cell J)