Worked examples — CXL (Compute Express Link)

6.3.5 · D3 · Hardware › Interconnects, Buses & SoC › CXL (Compute Express Link)

Ye page CXL (Compute Express Link) ka deep dive hai. Ye Cache-CoherencyProtocols, DMA-(Direct-Memory-Access), NUMA-(Non-Uniform-Memory-Access) aur HBM-(High-Bandwidth-Memory) ke ideas pe lean karta hai, aur Gen-Z-and-CCIX se contrast karta hai. Agar tum yahi material Hinglish mein chahte ho, to dekho 6.3.05 CXL (Compute Express Link) (Hinglish).

---

Scenario matrix

Koi bhi number aane se pehle, aao har tarah ke case ka naam rakhe jo ye topic tumpe throw kar sakta hai. Har row ko ek "question ka shape" samjho. Neeche har worked example us cell ke saath tagged hai jo woh fill karta hai.

#	Case class	Awkward part	Filled by
A	Bandwidth arithmetic (normal)	encoding overhead 128b/130b, GT/s → GB/s	Ex 1
B	Bandwidth — limiting / degenerate	×1 lane, aur PCIe 6.0 PAM-4 (no 130-bit tax)	Ex 2
C	CXL.cache — line hai Modified (dirty)	share karne se pehle write back karna padega	Ex 3
D	CXL.cache — line hai Invalid (cold miss)	memory se fetch karo, koi snoop hit nahi	Ex 4
E	CXL.cache — write intent (ReadOwn/RFO)	doosri copy ko invalidate karo, single-writer	Ex 5
E2	CXL.cache — Exclusive state	akela clean copy, silently Modified ho jaata hai	Ex 5b
F	CXL.mem — address routing (DRAM vs expander)	kaun sa transaction issue hoga? latency mix	Ex 6
G	Real-world word problem	AI model VRAM mein fit nahi — capacity math	Ex 7
H	Exam twist / trap	"kya CXL.io coherency maintain karta hai?" + pooling share math	Ex 8

Do terms jo hum baar baar use karenge, ek baar define kiye yahan taaki koi symbol unexplained na rahe:

---

Ex 1 — Bandwidth, normal case (cell A)

Forecast: padhne se pehle guess karo — kya ye 30, 60, ya 120 GB/s ke kareeb hoga ek taraf? Apna guess likho.

1. Per-lane raw rate. 32 GT/s × 1 bit/transfer = $32\times 10^9$ bits/s. Ye step kyun? PCIe 5.0 par ek transfer = ek bit, toh raw bit-rate = transfer-rate.
2. Encoding tax lagao. $\frac{128}{130}$ se multiply karo: $32\times 10^9\times \frac{128}{130}=31.507\times 10^9\text{bits/s}\approx 31.5\text{Gb/s}.$ Ye step kyun? Har 130 transmitted bits mein se sirf 128 tumhara data hain.
3. Bits → bytes. 8 se divide karo: $31.5/8=3.938\text{GB/s per lane}$ (decimal). Ye step kyun? Bandwidth bytes mein quote hoti hai; 8 bits = 1 byte.
4. 16 lanes tak scale karo. $16\times 3.938=63.0\text{GB/s one direction}$. Ye step kyun? Lanes parallel mein run karte hain aur linearly add hote hain.

Verify: 63 GB/s ek taraf → 126 GB/s aggregate exactly wahi "64 GB/s each way ≈ 128 GB/s" figure hai jo parent quote karta hai (rounded). Units check: $\frac{\text{transfers}}{\text{s}}\cdot \text{bit}\cdot \frac{\text{byte}}{8\text{bit}}=\text{bytes/s}$ ✓.

---

Ex 2 — Bandwidth extremes par (cell B)

Forecast: kya Gen-5 par ek lane ek poore SATA SSD (~0.55 GB/s) se zyada fast hogi? Roughly kitna zyada?

1. (a) Degenerate ×1. Ex 1 se, ek lane = 3.938 GB/s. Ye poore device ki bandwidth hai. Ye step kyun? ×1 sabse chhoti legal width hai — woh limiting case jahan "lanes se scale" 1 se multiply karta hai.
2. SATA se sanity check. $3.938/0.55\approx 7.2\times$ ek SATA SSD se zyada fast, sirf ek lane par. Ye step kyun? Confirm karta hai ki degenerate CXL link bhi slow nahi hota.
3. (b) PCIe 6.0 raw per lane. 64 GT/s × 2 bits/transfer (PAM-4 4 voltage levels bhejta hai = 2 bits) $=128\times 10^9$ bits/s. Ye step kyun? Gen-6 ne signalling bhi double kiya aur 2 bits per transfer pack kiya — isliye 6.0 ≈ 5.0 ke raw ka 4× hai.
4. Efficiency + bytes + 16 lanes. Framing efficiency lo $\approx \frac{242}{256}=0.945$: $128\times 10^9\times 0.945/8=15.12\text{GB/s per lane},\times 16=242\text{GB/s one way}.$ Ye step kyun? Ex 1 wali hi recipe, naye numbers ke saath. Aggregate ≈ 484 GB/s — parent ka "128 GB/s each way" CXL 3.0 ke liye zyada conservative rounding use karta hai, lekin Gen-5 se doubling wahi point hai.

Verify: Gen-6 one-way (242 GB/s) ÷ Gen-5 one-way (63 GB/s) $\approx 3.8\times$ — "roughly 4× per lane same lane count" ke saath consistent. Degenerate ×1 = 3.938 GB/s ✓.

---

Ex 3 — CXL.cache, line hai Modified (cell C)

Figure 1 (neeche) transaction dikhata hai: CPU cache (amber, 0x1000 ki dirty M copy hold kiye), DRAM box (grey, stale), aur GPU cache (cyan, abhi I). Amber arrows teen steps number karte hain — GPU ka request host ki taraf jaana, CPU ka write-back DRAM tak, aur data downgrade ke saath wapas dena. Agar image nahi dikhi: ye left-to-right flow hai, CPU left par, GPU right par, DRAM bottom-left par, end-states neeche print hain.

Forecast: kya GPU DRAM wali value lega ya CPU ki dirty value? CPU kaunsi state mein khatam hoga?

1. GPU request bhejta hai. ReadShared issue hota hai; host par ye CPU ke cache mein SnpData (snoop-for-data, upar define kiya) ban jaata hai. Ye step kyun? GPU assume nahi kar sakta ki DRAM fresh hai — usse host se poochna padega, kyunki coherency CPU ke home agent par rehti hai.
2. CPU apna cache snoop karta hai — hit, Modified. Look-up dirty line find karta hai (Figure 1 mein amber block). Ye step kyun? Modified matlab "DRAM stale hai; mere paas akeli sach copy hai." DRAM se serve karna yahan garbage return karta.
3. Memory mein write-back. CPU dirty line ko DRAM mein flush karta hai. Ye step kyun? Isse DRAM aur cache agree karte hain, taaki baad mein reader safely dono mein se koi bhi use kar sake.
4. Data deliver + downgrade. CPU line GPU ko bhejta hai aur M → S ho jaata hai. Ye step kyun? Ab do readers hain, toh single-writer rule force karta hai ki dono clean Shared ho jaayein.

Verify: end states — CPU = S, GPU = S, DRAM = fresh. Writers ginlo: 0 (single-writer-OR-multiple-reader invariant holds). GPU ki value CPU ki pre-request dirty value ke barabar hai, purani DRAM value nahi ✓.

---

Ex 4 — CXL.cache, cold Invalid miss (cell D)

Forecast: yahan kitne write-backs hote hain? Ex 3 se zyada ya kam?

1. GPU ReadShared(0x2000) bhejta hai. Host par ye phir SnpData ban jaata hai. Ye step kyun? Protocol entry point same hai — GPU pehle se nahi jaanta ki ye hit hoga ya nahi.
2. CPU snoop → miss (Invalid). Koi dirty owner nahi. Ye step kyun? Invalid matlab "kisi ke paas valid data nahi" — toh write back karne ke liye kuch hai hi nahi.
3. Home agent DRAM se fetch karta hai. DRAM already truth rakhta hai. Ye step kyun? Koi dirty copy nahi, toh DRAM authoritative hai — seedha padho.
4. Deliver + Shared install karo. GPU line S mein cache karta hai; CPU Invalid rehta hai (usse kabhi chahiye hi nahi tha). Ye step kyun? Sirf requester ko data chahiye; CPU ko force karke cache karwana uska cache waste karta.

Verify: write-backs = 0 (Ex 3 ke 1 ke mukable — degenerate-clean case sasta hai). End states: CPU I, GPU S, DRAM unchanged ✓.

---

Ex 5 — CXL.cache with write intent (ReadOwn / RFO) (cell E)

Figure 2 (neeche) write-intent path dikhata hai: CPU cache left par S se I tak drop hota hai (grey, hatched — matlab "invalidated"), GPU cache right par I se M tak rise karta hai (amber). Ek cyan arrow ReadOwn request left le jaata hai; ek amber arrow ownership right le aata hai. Caption neeche remind karta hai "writers = 1 (GPU in M)". Agar nahi dikha: ye wahi CPU/GPU two-box layout hai jaise Figure 1 mein, bina DRAM box ke.

Forecast: Ex 3 mein dono Shared khatam hue. Yahan GPU write karna chahta hai. Kya do Shared copies bach sakti hain? CPU ki final state guess karo.

1. GPU ReadOwn(0x1000) bhejta hai. Ye step kyun? Padhna kaafi nahi — likhne ke liye GPU ko sole owner banana padega, warna CPU ki stale copy linge rahegi.
2. CPU snoop → hit, Shared. CPU ke paas ek clean shared copy hai. Ye step kyun? Ek shared copy jo GPU overwrite karne wala hai usse deal karna padega, warna CPU baad mein stale data padhta.
3. CPU apni copy SnpInv se invalidate karta hai. CPU S → I ho jaata hai (Figure 2 mein grey/hatched block). Ye step kyun? Single-writer rule: agar GPU M hold karega, toh har doosri copy I honi chahiye. Ye crucial difference hai Ex 3 ke downgrade-to-Shared se — ek read SnpData trigger karta hai, ek write SnpInv trigger karta hai.
4. GPU Modified install karta hai. GPU line M mein cache karta hai aur ab freely likh sakta hai. Ye step kyun? Exclusive dirty ownership exactly wahi hai jo "modify karne ki intent" require karta hai.

Verify: writers = 1 (GPU, M mein), valid copies wale doosre readers = 0 → invariant holds. Contrast table:

Request	CPU was S →	GPU ends	Writers
ReadShared (Ex 3-style)	S	S	0
ReadOwn (this Ex)	I	M	1

Write-intent path invalidate karta hai na ki share ✓.

---

Ex 5b — CXL.cache, Exclusive state (cell E2)

Forecast: Ex 4 ne Shared install kiya — lekin woh isliye tha kyunki hum assume kar rahe the ki doosre bhi pad sakte hain. Agar GPU provably akela holder hai, kya Shared wasteful hai? Smarter state guess karo.

1. GPU ReadShared(0x3000) bhejta hai; host SnpData sab caches par issue karta hai → sab miss. Ye step kyun? Host ko confirm karna padega ki koi aur line nahi rakhta pehle, tabhi exclusivity promise kar sakta hai.
2. Home agent "no other sharers" dekhta hai aur line Exclusive ke roop mein return karta hai. GPU E install karta hai (clean, sole copy). Ye step kyun? Exactly iske liye Exclusive exist karta hai: ek clean copy jo kisi aur ke paas nahi. S install karna legal hoga lekin pessimistic.
3. GPU baad mein line write karta hai — silent upgrade E → M. Kyunki koi aur cache line nahi rakhta, koi SnpInv ki zaroorat nahi. Ye step kyun? E ka poora payoff yahi hai: Modified mein jaana zero bus traffic costa hai. Agar line Shared hoti, GPU ko pehle ReadOwn/SnpInv bhejna padta (jaise Ex 5 mein).
4. Ex 4 se contrast. Ex 4 ne S install kiya (ek possible second reader model kiya tha); yahan hum E install karte hain (provably sole holder). Dono cold reads hain — difference ye hai ki home agent koi doosra sharer report karta hai ya nahi. Ye step kyun? Ye woh fourth MESI state exercise karta hai jo koi doosra example touch nahi karta.

Verify: cold read with no other sharers → E install karo; baad mein write → E → M with 0 invalidations. Ex 5 ke S → M se compare karo jisme 1 SnpInv chahiye tha. Toh E exactly ek snoop message per first-write bachata hai ✓.

---

Ex 6 — CXL.mem address routing (cell F)

Figure 3 (neeche) ek address-space bar hai: ek horizontal ruler jo 512 GiB ceiling par ek amber vertical line se split hota hai. Line ke left mein sab cyan DRAM region hai; right mein sab amber CXL region hai. Neeche do access boxes apne regions ki taraf point karte hain — 0x40_0000_0000 (256 GiB) DRAM mein ~80 ns par, 0xA0_0000_0000 (640 GiB) CXL mein ~150 ns par — weighted-average result neeche print hai. Agar nahi dikh raha: ek number line imagine karo jahan address ki value akele decide karti hai DRAM vs CXL.

Forecast: hex mein 512 GiB hai 0x80_0000_0000. Do addresses mein se kaun usse upar hai?

1. DRAM ceiling compute karo. $512\text{GiB}=512\times 2^{30}$ bytes $=\text{0x80\_0000\_0000}$. Ye step kyun? Memory controller physical address ko is boundary se compare karta hai DRAM vs CXL decide karne ke liye. (Capacity/address math binary hai — convention box dekho.)
2. 0x40_0000_0000 classify karo. Woh $256\times 2^{30}$ bytes = 256 GiB hai — ceiling se neeche → DRAM, DDR command issue karo, ~80 ns. Ye step kyun? Local DRAM fast tier hai; koi CXL transaction form nahi hota.
3. 0xA0_0000_0000 classify karo. Woh $640\times 2^{30}$ bytes = 640 GiB hai — 512 GiB se upar → CXL.mem, MemRd ke roop mein format hoke expander ko route hota hai, ~150 ns. Ye step kyun? Ye NUMA idea hai: wahi load instruction alag time costa hai is baat par depend karte hue ki address kahaan rehta hai.
4. Mixed-workload average. Agar 70% accesses DRAM hit karte hain aur 30% CXL hit karte hain: $0.7\times 80+0.3\times 150=56+45=101\text{ns average}.$ Ye step kyun? Tiered memory tab hi fayda deta hai jab hot data DRAM mein rahe.

Verify: 0x40_0000_0000 = $256\times 2^{30}$ = 256 GiB < 512 GiB ✓; 0xA0_0000_0000 = $640\times 2^{30}$ = 640 GiB > 512 GiB ✓; weighted latency = 101 ns ✓. Note karo 150 ns abhi bhi NVMe (~µs) se ~$10\times$ fast hai, parent se match karta hai.

---

Ex 7 — Real-world word problem: bada model fit karna (cell G)

Forecast: 70 B params × 2 bytes — kya woh 80 GB se andar rahega ya bahar jaayega?

1. Weight footprint. $70\times 10^9\text{params}\times 2\text{bytes}=140\times 10^9$ bytes $=140\text{GB}$ (decimal). Ye step kyun? fp16 = 2 bytes each; count ko width se multiply karo. Hum decimal convention use karte hain kyunki model sizes hamesha usi tarah quote hoti hain.
2. (a) VRAM mein fit? $140\text{GB}>80\text{GB}$ → fit nahi hota. Ye HBM capacity wall hai. Ye step kyun? Exactly isliye hum CXL expander ki taraf jaate hain fast-but-small local VRAM ki jagah.
3. Weights CXL expander mein rakho. 256 GB expander mein se 140 GB use hota hai; GPU demand par CXL.cache ke zariye 64-byte lines pull karta hai. Ye step kyun? CXL GPU ko weights coherently dekhne deta hai bina manual DMA copy loop ke.
4. (b) Headroom. $256-140=116\text{GB}$ expander mein bacha KV-cache / activations ke liye. Ye step kyun? Long-context inference ko growing KV-cache chahiye; confirm karo ki fit hoga.

Verify: weights = 140 GB ✓, fit-in-80 GB = False ✓, expander headroom = 116 GB ✓. Units: params × (bytes/param) = bytes ✓.

---

Ex 8 — Exam twist / trap + pooling share (cell H)

Forecast: (a) padhne se pehle sub-protocol ka naam batao. (b) guess karo ki har host 100 GiB se zyada khoega ya kam.

1. (a) Trap khatam karo — verdict. Jhooth. CXL.io non-coherent PCIe-style I/O hai (config, MMIO, DMA) — ye sirf backward-compatibility path hai. Coherency aati hai CXL.cache se (device host memory cache karta hai) aur CXL.mem se (host device memory use karta hai). Ye step kyun? Exam trap "CXL link par run karta hai" ko "coherent hai" se confuse karta hai. Sirf .cache / .mem coherency carry karte hain; har device ko abhi bhi .io support karna padta hai, lekin .io support karna bilkul bhi coherency grant nahi karta.
2. (b) Original share. $1024\text{GiB}/4=256\text{GiB per host}$. Ye step kyun? Current chaar hosts mein even split.
3. (b) 5 hosts ke saath naya share. $1024\text{GiB}/5=204.8\text{GiB per host}$. Ye step kyun? fabric/pooling idea: total capacity fixed hai, toh zyada hosts = chhote slices.
4. (b) Har host ka drop. $256-204.8=51.2\text{GiB}$ khoya har ek ne — 100 GiB se kam. Ye step kyun? Sanity check: paanch mein se ek aur user add karna kisi bhi single host ko poore pool ÷ new count se zyada nahi costa, toh ~51 GiB loss range mein hai.

Verify: (a) .io-is-coherent = False; coherency = .cache + .mem ✓. (b) original = 256 GiB ✓, new = 204.8 GiB ✓, drop = 51.2 GiB ✓, aur 51.2 < 100 ✓.

---

Recall Quick self-test

Kaun sa snoop message line share karta hai, aur kaun invalidate karta hai? ::: SnpData share karta hai (read satisfy karta hai, clean copies chhod jaata hai); SnpInv invalidate karta hai (write intent ke liye). Kaun sa request CPU ko Shared tak downgrade karta hai, aur kaun Invalid tak? ::: ReadShared → CPU Shared ho jaata hai; ReadOwn/RFO → CPU Invalid ho jaata hai (single-writer). Ek cold ReadShared miss jisme KOI doosra sharer NAHI, kaun si state install karta hai aur kyun? ::: Exclusive (E) — akela clean copy, toh baad mein write E→M zero snoops se upgrade karta hai. Kya CXL.io cache coherency provide karta hai? ::: Nahi — woh CXL.cache aur CXL.mem ka kaam hai; CXL.io plain non-coherent PCIe hai. PCIe 5.0 ×16 usable one-way bandwidth? ::: ≈ 63 GB/s decimal (32 GT/s × 128/130 ÷ 8 × 16).