Worked examples — CXL (Compute Express Link)

This page is a deep dive of CXL (Compute Express Link). It leans on ideas from Cache-CoherencyProtocols, DMA-(Direct-Memory-Access), NUMA-(Non-Uniform-Memory-Access) and HBM-(High-Bandwidth-Memory), and contrasts with Gen-Z-and-CCIX. If you want the same material in Hinglish, see 6.3.05 CXL (Compute Express Link) (Hinglish).

---

The scenario matrix

Before any numbers, let us name every kind of case this topic can throw at you. Think of each row as a "shape of question". Every worked example below is tagged with the cell it fills.

#	Case class	The awkward part	Filled by
A	Bandwidth arithmetic (normal)	encoding overhead 128b/130b, GT/s → GB/s	Ex 1
B	Bandwidth — limiting / degenerate	×1 lane, and PCIe 6.0 PAM-4 (no 130-bit tax)	Ex 2
C	CXL.cache — line is Modified (dirty)	must write back before sharing	Ex 3
D	CXL.cache — line is Invalid (cold miss)	fetch from memory, no snoop hit	Ex 4
E	CXL.cache — write intent (ReadOwn/RFO)	invalidate the other copy, single-writer	Ex 5
E2	CXL.cache — Exclusive state	sole clean copy, silent upgrade to Modified	Ex 5b
F	CXL.mem — address routing (DRAM vs expander)	which transaction gets issued? latency mix	Ex 6
G	Real-world word problem	AI model too big for VRAM — capacity math	Ex 7
H	Exam twist / trap	"does CXL.io keep coherency?" + pooling share math	Ex 8

Two terms we will reuse, defined once here so no symbol is unearned:

---

Ex 1 — Bandwidth, the normal case (cell A)

Forecast: guess before reading — is it closer to 30, 60, or 120 GB/s one way? Write your guess down.

1. Per-lane raw rate. 32 GT/s × 1 bit/transfer = $32\times 10^9$ bits/s. Why this step? One transfer = one bit on PCIe 5.0, so raw bit-rate = transfer-rate.
2. Apply the encoding tax. Multiply by $\frac{128}{130}$: $32\times 10^9\times \frac{128}{130}=31.507\times 10^9\text{bits/s}\approx 31.5\text{Gb/s}.$ Why this step? Only 128 of every 130 transmitted bits are your data.
3. Bits → bytes. Divide by 8: $31.5/8=3.938\text{GB/s per lane}$ (decimal). Why this step? Bandwidth is quoted in bytes; 8 bits = 1 byte.
4. Scale to 16 lanes. $16\times 3.938=63.0\text{GB/s one direction}$. Why this step? Lanes run in parallel and add linearly.

Verify: 63 GB/s one way → 126 GB/s aggregate is exactly the "64 GB/s each way ≈ 128 GB/s" figure the parent quotes (rounded). Units check: $\frac{\text{transfers}}{\text{s}}\cdot \text{bit}\cdot \frac{\text{byte}}{8\text{bit}}=\text{bytes/s}$ ✓.

---

Ex 2 — Bandwidth at the extremes (cell B)

Forecast: will one lane at Gen-5 beat a whole SATA SSD (~0.55 GB/s)? By roughly how much?

1. (a) Degenerate ×1. From Ex 1, one lane = 3.938 GB/s. That's the whole device's bandwidth. Why this step? ×1 is the smallest legal width — the limiting case where "scale by lanes" multiplies by 1.
2. Sanity vs SATA. $3.938/0.55\approx 7.2\times$ faster than a SATA SSD, on a single lane. Why this step? Confirms even the degenerate CXL link is not slow.
3. (b) PCIe 6.0 raw per lane. 64 GT/s × 2 bits/transfer (PAM-4 sends 4 voltage levels = 2 bits) $=128\times 10^9$ bits/s. Why this step? Gen-6 doubled the signalling and packs 2 bits per transfer — this is why 6.0 ≈ 4× the raw of 5.0.
4. Efficiency + bytes + 16 lanes. Take framing efficiency $\approx \frac{242}{256}=0.945$: $128\times 10^9\times 0.945/8=15.12\text{GB/s per lane},\times 16=242\text{GB/s one way}.$ Why this step? Same recipe as Ex 1, new numbers. Aggregate ≈ 484 GB/s — the parent's "128 GB/s each way" for CXL 3.0 uses a more conservative rounding, but the doubling from Gen-5 is the point.

Verify: Gen-6 one-way (242 GB/s) ÷ Gen-5 one-way (63 GB/s) $\approx 3.8\times$ — consistent with "roughly 4× per lane and same lane count". Degenerate ×1 = 3.938 GB/s ✓.

---

Ex 3 — CXL.cache, line is Modified (cell C)

Figure 1 (below) shows the transaction: the CPU cache (amber, holding a dirty M copy of 0x1000), the DRAM box (grey, stale), and the GPU cache (cyan, currently I). Amber arrows number the three steps — the GPU's request going host-ward, the CPU's write-back down to DRAM, and the data being handed back with a downgrade. If you can't see the image: it is a left-to-right flow, CPU on the left, GPU on the right, DRAM bottom-left, with the end-states printed along the bottom.

Forecast: does the GPU get the DRAM value or the CPU's dirty value? What state does the CPU end in?

1. GPU sends the request. ReadShared is issued; at the host it becomes a SnpData (snoop-for-data, defined above) into the CPU's cache. Why this step? The GPU can't assume DRAM is fresh — it must ask the host, because coherency lives at the CPU's home agent.
2. CPU snoops its own cache — hit, Modified. The look-up finds a dirty line (amber block in Figure 1). Why this step? Modified means "DRAM is stale; I have the only true copy." Serving DRAM here would return garbage.
3. Write-back to memory. CPU flushes the dirty line to DRAM. Why this step? This makes DRAM and cache agree, so a later reader can safely use either.
4. Deliver data + downgrade. CPU sends the line to the GPU and drops M → S. Why this step? Two readers now exist, so the single-writer rule forces both to clean Shared.

Verify: end states — CPU = S, GPU = S, DRAM = fresh. Count the writers: 0 (single-writer-OR-multiple-reader invariant holds). The GPU's value equals the CPU's pre-request dirty value, not the old DRAM value ✓.

---

Ex 4 — CXL.cache, cold Invalid miss (cell D)

Forecast: how many write-backs happen here? More or fewer than Ex 3?

1. GPU sends ReadShared(0x2000). At the host this is again a SnpData. Why this step? Same protocol entry point — the GPU doesn't know in advance whether it's a hit.
2. CPU snoop → miss (Invalid). No dirty owner exists. Why this step? Invalid means "nobody has valid data" — so there is nothing to write back.
3. Home agent fetches from DRAM. DRAM already holds the truth. Why this step? With no dirty copy, DRAM is authoritative — read it directly.
4. Deliver + install Shared. GPU caches the line in S; CPU stays Invalid (it never wanted it). Why this step? Only the requester needs the data; forcing the CPU to cache it would waste its cache.

Verify: write-backs = 0 (vs 1 in Ex 3 — the degenerate-clean case is cheaper). End states: CPU I, GPU S, DRAM unchanged ✓.

---

Ex 5 — CXL.cache with write intent (ReadOwn / RFO) (cell E)

Figure 2 (below) depicts the write-intent path: CPU cache on the left drops from S to I (grey, hatched — meaning "invalidated"), GPU cache on the right rises from I to M (amber). A cyan arrow carries the ReadOwn request left; an amber arrow carries ownership back right. The caption at the bottom reminds you "writers = 1 (GPU in M)". If you can't see it: it is the same two-box CPU/GPU layout as Figure 1, minus the DRAM box.

Forecast: in Ex 3 both ended Shared. Here the GPU wants to write. Can two Shared copies survive? Guess the CPU's final state.

1. GPU sends ReadOwn(0x1000). Why this step? Reading isn't enough — to write, the GPU must become the sole owner, or a stale CPU copy would linger.
2. CPU snoop → hit, Shared. CPU has a clean shared copy. Why this step? A shared copy the GPU is about to overwrite must be dealt with, or the CPU would read stale data later.
3. CPU invalidates its copy via SnpInv. CPU drops S → I (grey/hatched block in Figure 2). Why this step? The single-writer rule: if the GPU will hold M, every other copy must be I. This is the crucial difference from Ex 3's downgrade-to-Shared — a read triggers SnpData, a write triggers SnpInv.
4. GPU installs Modified. GPU caches the line M and may now write freely. Why this step? Exclusive dirty ownership is exactly what "intent to modify" requires.

Verify: writers = 1 (GPU, in M), readers with valid copies = 0 others → invariant holds. Contrast table:

Request	CPU was S →	GPU ends	Writers
ReadShared (Ex 3-style)	S	S	0
ReadOwn (this Ex)	I	M	1

The write-intent path invalidates rather than shares ✓.

---

Ex 5b — CXL.cache, the Exclusive state (cell E2)

Forecast: Ex 4 installed Shared — but that was because we assumed others might read too. If the GPU is provably the only holder, is Shared wasteful? Guess the smarter state.

1. GPU sends ReadShared(0x3000); host issues SnpData to all caches → all miss. Why this step? The host must confirm nobody else holds the line before it can promise exclusivity.
2. Home agent sees "no other sharers" and returns the line as Exclusive. GPU installs E (clean, sole copy). Why this step? This is exactly what Exclusive exists for: a clean copy that nobody else has. Installing S instead would be legal but pessimistic.
3. GPU later writes the line — silent upgrade E → M. Because no other cache holds the line, no SnpInv is needed. Why this step? The whole payoff of E: the transition to Modified costs zero bus traffic. Had the line been Shared, the GPU would have had to send a ReadOwn/SnpInv first (as in Ex 5).
4. Contrast with Ex 4. Ex 4 installed S (it modelled a possible second reader); here we install E (provably sole holder). Both are cold reads — the difference is whether the home agent reports any other sharer. Why this step? This exercises the fourth MESI state that no other example touches.

Verify: cold read with no other sharers → install E; subsequent write → E → M with 0 invalidations. Compare to Ex 5's S → M which needed 1 SnpInv. So E saves exactly one snoop message per first-write ✓.

---

Ex 6 — CXL.mem address routing (cell F)

Figure 3 (below) is an address-space bar: a horizontal ruler split by an amber vertical line at the 512 GiB ceiling. Everything left of the line is the cyan DRAM region; everything right is the amber CXL region. Two access boxes below point up to their regions — 0x40_0000_0000 (256 GiB) into DRAM at ~80 ns, 0xA0_0000_0000 (640 GiB) into CXL at ~150 ns — with the weighted-average result printed at the bottom. If you can't see it: picture a number line where an address's value alone decides DRAM vs CXL.

Forecast: 512 GiB in hex is 0x80_0000_0000. Which of the two addresses is above that line?

1. Compute the DRAM ceiling. $512\text{GiB}=512\times 2^{30}$ bytes $=\text{0x80\_0000\_0000}$. Why this step? The memory controller compares the physical address against this boundary to decide DRAM vs CXL. (Capacity/address math is binary — see the convention box.)
2. Classify 0x40_0000_0000. That's $256\times 2^{30}$ bytes = 256 GiB — below the ceiling → DRAM, issue a DDR command, ~80 ns. Why this step? Local DRAM is the fast tier; no CXL transaction is formed.
3. Classify 0xA0_0000_0000. That's $640\times 2^{30}$ bytes = 640 GiB — above 512 GiB → CXL.mem, formatted as a MemRd and routed to the expander, ~150 ns. Why this step? This is the NUMA idea: the same load instruction costs different time depending on where the address lives.
4. Mixed-workload average. If 70% of accesses hit DRAM and 30% hit CXL: $0.7\times 80+0.3\times 150=56+45=101\text{ns average}.$ Why this step? Tiered memory only pays off if the hot data stays in DRAM.

Verify: 0x40_0000_0000 = $256\times 2^{30}$ = 256 GiB < 512 GiB ✓; 0xA0_0000_0000 = $640\times 2^{30}$ = 640 GiB > 512 GiB ✓; weighted latency = 101 ns ✓. Note 150 ns is still ~$10\times$ faster than NVMe (~µs), matching the parent.

---

Ex 7 — Real-world word problem: fitting a big model (cell G)

Forecast: 70 B params × 2 bytes — will that clear or blow past 80 GB?

1. Weight footprint. $70\times 10^9\text{params}\times 2\text{bytes}=140\times 10^9$ bytes $=140\text{GB}$ (decimal). Why this step? fp16 = 2 bytes each; multiply count by width. We use the decimal convention because model sizes are always quoted that way.
2. (a) Fit in VRAM? $140\text{GB}>80\text{GB}$ → does not fit. This is the HBM capacity wall. Why this step? This is precisely why we reach for a CXL expander instead of the fast-but-small local VRAM.
3. Place weights in the CXL expander. 140 GB of the 256 GB expander is used; the GPU pulls 64-byte lines on demand via CXL.cache. Why this step? CXL lets the GPU see the weights coherently without a manual DMA copy loop.
4. (b) Headroom. $256-140=116\text{GB}$ left in the expander for KV-cache / activations. Why this step? Long-context inference needs growing KV-cache; confirm it fits.

Verify: weights = 140 GB ✓, fit-in-80 GB = False ✓, expander headroom = 116 GB ✓. Units: params × (bytes/param) = bytes ✓.

---

Ex 8 — Exam twist / trap + pooling share (cell H)

Forecast: name the sub-protocol before reading (a). For (b), guess whether each host loses more or less than 100 GiB.

1. (a) Kill the trap — the verdict. False. CXL.io is non-coherent PCIe-style I/O (config, MMIO, DMA) — it is the backward-compatibility path only. Coherency comes from CXL.cache (device caches host memory) and CXL.mem (host uses device memory). Why this step? The exam trap conflates "runs over the CXL link" with "is coherent." Only .cache / .mem carry coherency; every device must still support .io, but supporting .io grants no coherency at all.
2. (b) Original share. $1024\text{GiB}/4=256\text{GiB per host}$. Why this step? Even split across the current four hosts.
3. (b) New share with 5 hosts. $1024\text{GiB}/5=204.8\text{GiB per host}$. Why this step? The fabric/pooling idea: total capacity is fixed, so more hosts = smaller slices.
4. (b) Drop per host. $256-204.8=51.2\text{GiB}$ lost each — less than 100 GiB. Why this step? Sanity check: adding one more of five users cannot cost any single host more than the whole pool ÷ new count, so a ~51 GiB loss is in range.

Verify: (a) .io-is-coherent = False; coherency = .cache + .mem ✓. (b) original = 256 GiB ✓, new = 204.8 GiB ✓, drop = 51.2 GiB ✓, and 51.2 < 100 ✓.

---

Recall Quick self-test

Which snoop message shares a line, and which invalidates it? ::: SnpData shares (satisfies a read, leaves clean copies); SnpInv invalidates (for write intent). Which request downgrades the CPU to Shared, and which to Invalid? ::: ReadShared → CPU goes to Shared; ReadOwn/RFO → CPU goes to Invalid (single-writer). A cold ReadShared miss with NO other sharers installs which state, and why? ::: Exclusive (E) — sole clean copy, so a later write upgrades E→M with zero snoops. Does CXL.io provide cache coherency? ::: No — that's CXL.cache and CXL.mem; CXL.io is plain non-coherent PCIe. PCIe 5.0 ×16 usable one-way bandwidth? ::: ≈ 63 GB/s decimal (32 GT/s × 128/130 ÷ 8 × 16).