6.3.5 · D1Interconnects, Buses & SoC

Foundations — CXL (Compute Express Link)

2,164 words10 min readBack to topic

This page assumes you know nothing. We build every word the CXL (Compute Express Link) parent note throws at you — cache, coherency, MESI, PCIe, GT/s, snoop, cache line, address space — starting from a picture of a chip talking to memory.


0. The starting picture: a processor and its memory

Before any fancy word, picture two boxes and a wire.

Figure — CXL (Compute Express Link)

Everything in CXL is about how many boxes there are, and how they agree on what's in the mailboxes.


1. Address — the mailbox number

Picture: a long row of numbered lockers. Address = locker number.

Why the topic needs it: the parent note writes things like MemRd to address 0x2_000_0000. That is nothing more than "fetch the data in locker number two-hundred-something-million".

Why CXL cares: CXL memory expanders let the processor use locker numbers beyond the ones served by its own local RAM. The lockers physically live in another box, but the processor names them the same way — see the figure.

Figure — CXL (Compute Express Link)
0x means "the number after me is in hexadecimal (base-16)"
it is just a counting shorthand engineers use; 0x1000 is the decimal number 4096.

2. Cache — a tiny fast notebook next to the worker

Reaching all the way to the memory wall is slow. So each processor keeps a small, very fast copy of the data it uses most, right next to it.

Picture: the worker keeps a sticky note of the numbers they're using. Reading the sticky note is instant; walking to the locker wall is slow.

Why the topic needs it: the parent says "CPU has a 64-byte cache line at address 0x1000". That is one sticky note holding the 64 bytes starting at locker 4096.


3. Coherency — everyone agrees on the newest value

Picture: the Google-Docs auto-sync. Two people edit the same document; the instant one types, the other sees it. That instant sync = coherency.

  • Coherent = all copies agree, always. ✅
  • Incoherent = copies disagree; someone is reading old data. ❌ (the "stale data" bug in the parent note.)

See Cache-CoherencyProtocols for the full machinery; here we only need the idea.


4. The four states: M, E, S, I (MESI)

To enforce coherency, each sticky-note copy carries a label describing how trustworthy and how private it is. There are exactly four labels.

Figure — CXL (Compute Express Link)

The golden rule these states enforce: single-writer OR multiple-readers. Either exactly one holder may write (M or E), or many may read but none may write (S). Never two writers.

Why the topic needs it: every CXL.cache example is a walk through these four labels — a locker goes M → S when a GPU asks to read it, etc. Once you know the labels, that walk is obvious.

Which state means "my copy is dirty and memory is stale"?
Modified (M)
Which state lets many caches hold the same line for reading?
Shared (S)

5. Snoop — "hey, do you have this locker?"

How does one processor find out whether another has a private, possibly-dirty copy? It asks. That question is called a snoop.

Picture: the worker holds up a locker number and shouts across the room, "Anyone holding #57? If it's dirty, give it here!"

Why the topic needs it: the parent's opcodes SnpData, SnpInv are exactly these shouts — Snp = snoop, Data = "give me the data", Inv = "invalidate your copy". A snoop-based coherency protocol is just this shouting done in hardware at nanosecond speed.


6. The wire: PCIe, lanes, and GT/s

Now the physical wire between the boxes. CXL does not invent a new cable — it borrows the one that GPUs already use: PCIe.

Picture: a lane is a single-file conveyor belt for bits. A ×16 link is 16 belts side by side, moving 16× the bits at once.

Why divide by 130 and not subtract?
The overhead is a fraction of the stream, so it scales the rate multiplicatively, not by a fixed subtraction.

Picture: one conveyor belt, but boxes of three colours take turns riding it. Same belt, shared cleverly.


7. The three CXL sub-protocols in one glance

With the vocabulary above, the parent's three protocols collapse to one picture: who is caching whose memory.

Figure — CXL (Compute Express Link)

Two related standards worth knowing: DMA-(Direct-Memory-Access) (moving data without bothering the CPU), and the pre-CXL rivals Gen-Z-and-CCIX. CXL's memory-expansion angle relates to HBM-(High-Bandwidth-Memory) and to NUMA-(Non-Uniform-Memory-Access), where different memory is at different distances/latencies — exactly the world CXL manages.


8. How these foundations feed the topic

Processor and Memory

Address and Address Space

Cache and Cache Line

CXL.mem grows address space

Cache Coherency problem

MESI states M E S I

Snoop requests SnpData SnpInv

CXL.cache coherent accelerator

PCIe lanes and GT per s

Bandwidth and Flex Bus

CXL runs on PCIe

CXL topic

Read it top-down: the plain picture of processor + memory splits into two families — addressing/caching (left) and the physical wire (right) — and both must be understood before the CXL topic at the bottom makes sense.


Equipment checklist

Test yourself — you're ready for the parent note when you can answer each without peeking.

What is an address, in one sentence?
The number of a mailbox/locker in memory that the processor names to fetch data.
What is a cache line and how big is it typically?
A fixed chunk of memory (usually 64 bytes) that moves between memory and cache as one unit.
State the golden coherency rule.
Single-writer OR multiple-readers — never two writers of the same location at once.
What do the letters M, E, S, I stand for?
Modified, Exclusive, Shared, Invalid.
Which MESI state means "my copy is dirty and memory itself is stale"?
Modified (M).
What does a snoop do?
Broadcasts "do you have this address, and in what state?" so caches hand over data or downgrade their labels.
What is one PCIe lane, and what does ×16 mean?
A lane is one serial bit-stream; ×16 means 16 parallel lanes working together.
Why is a PCIe 5.0 lane only ~3.94 GB/s despite 32 GT/s?
Because 128b/130b encoding spends 2 of every 130 bits on packaging, and 8 bits make a byte.
Name the three CXL sub-protocols and who caches whose memory.
CXL.io (plain PCIe), CXL.cache (accelerator caches CPU memory), CXL.mem (CPU uses device memory).
What does Flex Bus / time-multiplexing achieve?
It lets all three sub-protocols share the same physical PCIe lanes by taking turns.