CXL (Compute Express Link): Visual walkthrough — Hardware deep dive

We will lean on ideas from CXL (Compute Express Link) (the parent), and touch Cache-CoherencyProtocols, DMA-(Direct-Memory-Access), HBM-(High-Bandwidth-Memory), NUMA-(Non-Uniform-Memory-Access) and Gen-Z-and-CCIX where they help.

Step 1 — A wire that flips: what "transfer" even means

WHAT. A CXL link is, at the very bottom, a pair of copper wires — a lane. One wire carries a voltage that the sender flips between "high" and "low". The receiver, a fixed distance away, samples that voltage on a clock tick and calls it a 1 or a 0.

WHY start here. Every bandwidth number in the parent table is just "how many of these flips per second, minus the flips we waste on error-checking." If you understand one wire flipping, you understand the whole table. Nothing else is needed.

PICTURE. Look at the square wave below. Each vertical jump is the wire changing state. The little clock ticks underneath are the moments the receiver looks.

Step 2 — PCIe 5.0's raw rate: 32 GT/s per lane

WHAT. The PCIe 5.0 electrical spec pins the wire's flip rate at $32 GT/s$ . That is the raw number every CXL 1.0/2.0 link starts from.

WHY this number and not "just faster". The rate is capped by physics: push the flips faster and the copper smears the sharp edges (the signal blurs, high and low bleed together). $32 GT/s$ is the fastest the standard trusts the receiver to still tell high from low over a real motherboard trace. CXL reuses this exact electrical layer — that is the whole point, it rides PCIe's wire (contrast Gen-Z-and-CCIX, which tried to define new fabrics).

PICTURE. Same wire as Step 1, but now the ticks are packed $32$ billion to the second. The blur at the edges is why we can't simply crank higher.

Step 3 — The tax: 128b/130b encoding

WHAT. The sender does not send your 128 bits alone. It bolts on 2 extra bits (a "sync header") for every 128 bits of your data, making a 130-bit chunk. So out of every 130 transfers, only 128 carry your data.

WHY these 2 bits. The receiver must know where a chunk starts and must catch bit-flips from noise. Those 2 header bits mark the boundary and let the receiver re-lock its clock. It is a tax you pay so the link stays reliable — cheaper than the old PCIe 3.0 "8b/10b" scheme, which taxed 2 bits out of every 10 (20%!).

PICTURE. The bar below is one 130-bit chunk. The wide pale-yellow part is your payload; the thin pink sliver on the end is the 2 overhead bits.

Step 4 — One lane's real data rate

WHAT. Multiply the raw rate by the efficiency to get the useful data rate of a single lane.

WHY multiply. Bandwidth is "useful bits per second" = (transfers per second) × (fraction that are useful). That is literally $R_{raw} \times η$ . No other operation makes sense — we are just discounting the raw rate by the tax.

PICTURE. Two stacked bars: the tall one is $32$ raw, the shorter one below is what survives the tax, $31.5$ .

Step 5 — Widen the road: ×16 lanes

WHAT. A CXL link bundles many lanes side by side. A "×16" link is 16 identical lanes carrying data in parallel, in one direction. Add them up.

WHY it's just addition. The lanes are independent wires flipping at the same time — no shared bottleneck — so total data rate is simply 16 copies. This is the same "parallel lanes" trick HBM-(High-Bandwidth-Memory) uses internally, just over copper instead of a silicon interposer.

PICTURE. Sixteen parallel arrows, each $3.94 GB/s$ , merging into one fat highway labelled $63 GB/s$ .

Step 6 — Both directions: the parent's "64 GB/s"

WHAT. CXL wires are full-duplex: each lane is actually two wire-pairs, one for send, one for receive, running at the same time. So you get the ×16 rate each way, simultaneously.

WHY count both. Send and receive don't compete — separate copper. Marketing tables usually quote the sum ("aggregate"), which is where the parent's " $64 GB/s$ ( $32 + 32$ )" comes from — each direction rounded to $\approx 32 GB/s$ .

PICTURE. Two fat highways side by side, opposite arrows, each ~ $32 GB/s$ , summing to ~ $64$ .

Step 7 — The degenerate cases (never hit a surprise)

WHAT. What happens at the edges — a narrow link, an idle link, a link where CXL borrows lanes for coherency?

WHY cover them. The formula $B = N_{lanes} \times R_{raw} \times η$ must still give the right answer when inputs get small or weird, or you'll misread a spec sheet.

×1 link ( $N = 1$ ): $B = 3.94 GB/s$ one way. The formula degrades smoothly — a single lane is the base unit.
×0 / no lanes trained: link fails to come up, $B = 0$ . There is no "half a lane"; lane counts are $1, 2, 4, 8, 16$ only.
Idle link: the wire still flips (it sends "idle" filler symbols to keep the clock locked), so raw transfers continue at $32 GT/s$ , but useful bandwidth is $0$ — a reminder that GT/s ≠ delivered data.
Coherency traffic (CXL.cache/.mem) shares the same lanes: the Flex Bus time-multiplexes protocols, so if half the link's time slots carry snoop/coherency messages, your data throughput is roughly halved even though $B_{agg}$ is unchanged. The pipe's width is fixed; what flows through it is a scheduling choice.

PICTURE. A tick-strip showing the same fixed-width link with three fills: full data, idle filler, and a mix of data + coherency snoops.

The one-picture summary

The whole derivation on one board: one wire → 32 GT/s → tax it by 128/130 → 3.94 GB/s per lane → ×16 lanes → 63 GB/s one way → ×2 directions → 64 GB/s aggregate, with the coherency tax siphoning off some of that flow.

Recall Feynman retelling — say it out loud

Picture a single copper wire flicking a light on and off. PCIe 5.0 flicks it 32 billion times a second — that's the "raw" speed, one transfer per flick. But every 130 flicks, 2 are wasted on a little "are-you-still-there?" signal, so only 128/130 ≈ 98.5% is real data. That leaves about 31.5 gigabits, and since 8 bits make a byte, one wire carries about 3.94 gigabytes per second. Now lay 16 such wires side by side (a ×16 link): 16 × 3.94 ≈ 63 GB/s heading one way. And because CXL has separate send and receive wires, you get that speed both ways at once — sum it and round, and you land on the famous ~64 GB/s. That's the entire parent-note formula, built from a wiggling wire. The catch: CXL runs cache-coherency chatter over those very same lanes, so some of that 64 is spent keeping the CPU and accelerator in sync — which is exactly the magic PCIe alone couldn't do.

Recall Quick self-test

Raw PCIe 5.0 per-lane transfer rate? ::: 32 GT/s The encoding efficiency fraction? ::: 128/130 ≈ 0.985 Usable data rate of one lane in GB/s? ::: about 3.94 GB/s Why multiply by 128/130 and not 130/128? ::: only 128 of every 130 transfers carry your data, so you keep a fraction less than 1 Where does CXL 3.0's jump to 128 GB/s come from? ::: it rides PCIe 6.0, which packs ~2× the data per transfer (PAM4 signalling) Does CXL give more raw GB/s than PCIe 5.0? ::: no — same wire, same bandwidth; CXL's win is coherency and lower latency