Exercises — Infinity Fabric - mesh interconnects

• (a) shared bus → (ii). Saare cores ek wire tap karte hain, isliye ek waqt mein sirf ek talker aur har extra core capacitance add karta hai jo safe clock ko low karta hai.
• (b) ring → (i). Average trip loop ka quarter hota hai, isliye hops $=N/4$ node count ke saath linearly badhte hain.
• (c) mesh → (iii). Distances $\sqrt{N}$ jaisi badhti hain kyunki tum $N$ nodes ko $\sqrt{N}\times \sqrt{N}$ square mein fold karte ho — bahut slow growth.

5 ports: chaar cardinal directions — North, South, East, West — plus ek local port jo router ko uske apne agent (core + L2 slice jo wo serve karta hai) se connect karta hai. Chaar neighbours se baat karne ke liye, ek "home" se baat karne ke liye.

Kya karte hain: bytes-per-cycle × cycles-per-second. $B{W}_{\text{1-dir}}=32\text{B}\times 1.8\times 10^9\text{cycle/s}=57.6\text{GB/s}$ Kyun multiply karte hain: har tick 32 bytes move karta hai; $1.8\times 10^9$ ticks per second hain, isliye total bytes/second unka product hai. Bidirectional (ek aur identical lane doosri taraf run karta hai): $B{W}_{\text{bi}}=2\times 57.6=115.2\text{GB/s}$

X phase: $x=1$ se $x=3$ tak → $\Delta x=|3-1|=2$ hops east: $(1,0)\to (2,0)\to (3,0)$. Y phase: $y=0$ se $y=3$ tak → $\Delta y=|3-0|=3$ hops north: $(3,0)\to (3,1)\to (3,2)\to (3,3)$. $\text{hops}=|\Delta x|+|\Delta y|=2+3=5$ Visited path: $(1,0),(2,0),(3,0),(3,1),(3,2),(3,3)$.

Ring: hops $=N/4=16/4=4$, isliye $T_{\text{ring}}=4\times 2=8$ cycles. Mesh (exact): $k=\sqrt{N}=4$ ke saath, use karo ${\text{hops}}_{\text{mesh}}=\frac{2(N-1)}{3\sqrt{N}}=\frac{2\cdot 15}{3\cdot 4}=\frac{30}{12}=2.5$ hops (approximate $\frac{2\sqrt{N}}{3}=2.667$ dropped $\frac{-1}{k}$ term se over-count karta). Isliye $T_{\text{mesh}}=2.5\times 2=5.0$ cycles. Speedup: $\frac{T_{\text{ring}}}{T_{\text{mesh}}}=\frac{8}{5.0}=1.6\times$ Mesh is size pe roughly 1.6× lower latency hai — aur gap badhta hai jab $N$ badhta hai kyunki ring $N$ ki tarah scale karta hai jabki mesh $\sqrt{N}$ ki tarah.

Bisection = wo kam se kam links jo tumhe cut karne padenge network ko do equal halves mein separate karne ke liye; yeh traffic ko cap karta hai jo halves ke beech flow kar sakta hai.

• Ring: loop ko slice karna exactly 2 links cross karta hai (loop do jagah cut hota hai).
• Mesh (4×4): do columns ke beech cut karo — woh horizontal links ka ek poora column sever karta hai $=\sqrt{N}=4$ links. $\frac{4}{2}=2\times \text{ higher bisection mesh ke liye}$

20 ns ko fabric cycles mein convert karo: ek cycle $1/(1.8\times 10^9)\text{s}\approx 0.5556\text{ns}$ rehta hai. $\frac{20\text{ns}}{0.5556\text{ns/cycle}}=36\text{cycles per crossing}$ Ek read ke liye crossings count karo (round trip): request path = 2 crossings, reply path = 2 crossings → 4 crossings total. $4\times 36=144\text{cycles}$ Intra-chiplet lookup add karo: $144+5=149$ cycles full request-to-data round trip ke liye. Insight: fabric crossings, cache lookup nahi, dominate karte hain — aur kyunki read wapas aana chahiye, return path crossing count double kar deta hai. Exactly issi liye AMD memory controllers ek central I/O die pe rakhta hai taaki har chiplet symmetric ho (relate karta hai 6.2.8-NUMA-Architecture non-uniform access se). Note: parent note ka "~75 cycles" sirf one-way request path quote karta hai (2 crossings + lookup ≈ 77); full read latency roughly double hai.

Broadcast: request har core ko jaati hai → $N-1=63$ probes (writer ke siwa sab). Directory: home L3 slice exact sharer list jaanta hai, isliye sirf $k=3$ holders ko probe karta hai (ya $k-1=2$ agar writer already hold karta hai; statement ke hisaab se $k=3$ use karo). $\frac{63}{3}=21\times \text{ kam probe traffic}$ Kyun directory scale karta hai: broadcast har write pe $O(N)$ hai; directory $O(k)$ hai jahan $k$ typically 1–3 hota hai, core count se independent. Yahi AMD ke Scalable Coherent Protocol ki core baat hai — dekho 6.3.5-Cache-Coherency-Protocols.

Step 1 — memory bandwidth. DDR "MT/s" dono edges count karta hai, isliye $6000\text{MT/s}=6.0\times 10^9$ transfers/s per channel. $B{W}_{\text{mem}}=6.0\times 10^9\times 8\text{B}\times 2\text{channels}=96\text{GB/s}$ Step 2 — required FCLK. Fabric 32 B/cycle move karta hai; set karo $32\times \text{FCLK}\ge 96\times 10^9$: $\text{FCLK}\ge \frac{96\times 10^9}{32}=3.0\times 10^9=3.0\text{GHz}$ Step 3 — ratio. MCLK (memory clock) MT/s rate ka aadha hai: $6000/2=3000\text{MHz}=3.0\text{GHz}$. Isliye balanced ratio hai: $\text{FCLK}:\text{MCLK}=3.0:3.0=1:1$ Insight: 1:1 FCLK:MCLK ratio exactly isliye enthusiasts FCLK ko memory speed track karne ke liye tune karte hain — neeche giro aur fabric cores ko woh bandwidth bhookha rakh deta hai jo DIMMs supply kar sakti hain.

Perfect squares ko directly exact mesh formula $\frac{2(N-1)}{3\sqrt{N}}$ vs ring $N/4$ se test karo:

• $N=4$: mesh $=\frac{2\cdot 3}{3\cdot 2}=1.0$, ring $=1.0$ → tie, mesh strictly less nahi.
• $N=9$: mesh $=\frac{2\cdot 8}{3\cdot 3}=\frac{16}{9}\approx 1.78$, ring $=2.25$ → mesh strictly jeet ta hai. $\boxed{N=9}\text{ sabse chota square hai jahan mesh strictly ring ko beat karta hai.}$ Yeh real-world "~10–12 core" crossover se match karta hai jab tum routing aur area overheads add karo jo practical break-even ko pure-math $N=9$ se thoda zyada push karte hain. Short mein: mathematics $N=9$ pe mesh ke favour mein tip karta hai, lekin bade routers banane ki messy engineering realities asli switch ko roughly ek dozen cores tak delay karti hain.

Bus weakness ::: ek waqt mein ek transaction + electrical loading (ek shared wire). N nodes ke liye ring average hops ::: N/4 (exact). k×k grid ke liye exact mesh average hops ::: 2(N−1)/(3√N), jo bade N ke liye 2√N/3 → ho jaata hai. Kyun directory broadcast snooping ko beat karta hai ::: O(k) targeted probes vs O(N) broadcast, k = sharers. Ek read ki worst-case latency kitne fabric crossings count karti hai ::: chaar (request out + data reply, har ek do crossings). Balanced FCLK:MCLK ratio ::: 1:1 taaki fabric bandwidth memory bandwidth se match kare. Sabse chota square N jahan mesh strictly ring ko beat karta hai ::: N = 9.