Exercises — Infinity Fabric - mesh interconnects

• (a) shared bus → (ii). All cores tap one wire, so only one talker at a time and each extra core adds capacitance that lowers the safe clock.
• (b) ring → (i). Average trip is a quarter of the loop, so hops $=N/4$ grow linearly with node count.
• (c) mesh → (iii). Distances grow like $\sqrt{N}$ because you fold $N$ nodes into a $\sqrt{N}\times \sqrt{N}$ square — much slower growth.

5 ports: four cardinal directions — North, South, East, West — plus one local port that connects the router to its own agent (the core + L2 slice it serves). Four to talk to neighbours, one to talk to "home".

What we do: bytes-per-cycle × cycles-per-second. $B{W}_{\text{1-dir}}=32\text{B}\times 1.8\times 10^9\text{cycle/s}=57.6\text{GB/s}$ Why multiply: each tick moves 32 bytes; there are $1.8\times 10^9$ ticks per second, so total bytes/second is the product. Bidirectional (a second identical lane runs the other way): $B{W}_{\text{bi}}=2\times 57.6=115.2\text{GB/s}$

X phase: from $x=1$ to $x=3$ → $\Delta x=|3-1|=2$ hops east: $(1,0)\to (2,0)\to (3,0)$. Y phase: from $y=0$ to $y=3$ → $\Delta y=|3-0|=3$ hops north: $(3,0)\to (3,1)\to (3,2)\to (3,3)$. $\text{hops}=|\Delta x|+|\Delta y|=2+3=5$ Path visited: $(1,0),(2,0),(3,0),(3,1),(3,2),(3,3)$.

Ring: hops $=N/4=16/4=4$, so $T_{\text{ring}}=4\times 2=8$ cycles. Mesh (exact): with $k=\sqrt{N}=4$, use ${\text{hops}}_{\text{mesh}}=\frac{2(N-1)}{3\sqrt{N}}=\frac{2\cdot 15}{3\cdot 4}=\frac{30}{12}=2.5$ hops (the approximate $\frac{2\sqrt{N}}{3}=2.667$ would over-count by the dropped $\frac{-1}{k}$ term). So $T_{\text{mesh}}=2.5\times 2=5.0$ cycles. Speedup: $\frac{T_{\text{ring}}}{T_{\text{mesh}}}=\frac{8}{5.0}=1.6\times$ The mesh is about 1.6× lower latency at this size — and the gap widens as $N$ grows because the ring scales like $N$ while the mesh scales like $\sqrt{N}$.

Bisection = fewest links you must cut to separate the network into two equal halves; it caps the traffic that can flow between the halves.

• Ring: slicing a loop crosses exactly 2 links (the loop is cut in two places).
• Mesh (4×4): cut between two columns — that severs one full column of horizontal links $=\sqrt{N}=4$ links. $\frac{4}{2}=2\times \text{ higher bisection for the mesh}$

Convert 20 ns to fabric cycles: one cycle lasts $1/(1.8\times 10^9)\text{s}\approx 0.5556\text{ns}$. $\frac{20\text{ns}}{0.5556\text{ns/cycle}}=36\text{cycles per crossing}$ Count the crossings for a read (round trip): request path = 2 crossings, reply path = 2 crossings → 4 crossings total. $4\times 36=144\text{cycles}$ Add intra-chiplet lookup: $144+5=149$ cycles for the full request-to-data round trip. Insight: the fabric crossings, not the cache lookup, dominate — and because a read must come back, the return path doubles the crossing count. This is exactly why AMD keeps the memory controllers on a central I/O die so every chiplet is symmetric (relates to 6.2.8-NUMA-Architecture non-uniform access). Note: the parent note's "~75 cycles" quotes only the one-way request path (2 crossings + lookup ≈ 77); the full read latency is roughly double that.

Broadcast: the request goes to every core → $N-1=63$ probes (all except the writer). Directory: the home L3 slice knows the exact sharer list, so it probes only the $k=3$ holders (or $k-1=2$ if the writer already holds it; use $k=3$ per statement). $\frac{63}{3}=21\times \text{ less probe traffic}$ Why directory scales: broadcast is $O(N)$ per write; directory is $O(k)$ with $k$ typically 1–3, independent of core count. This is the heart of AMD's Scalable Coherent Protocol — see 6.3.5-Cache-Coherency-Protocols.

Step 1 — memory bandwidth. DDR "MT/s" already counts both edges, so $6000\text{MT/s}=6.0\times 10^9$ transfers/s per channel. $B{W}_{\text{mem}}=6.0\times 10^9\times 8\text{B}\times 2\text{channels}=96\text{GB/s}$ Step 2 — required FCLK. Fabric moves 32 B/cycle; set $32\times \text{FCLK}\ge 96\times 10^9$: $\text{FCLK}\ge \frac{96\times 10^9}{32}=3.0\times 10^9=3.0\text{GHz}$ Step 3 — the ratio. MCLK (the memory clock) is half the MT/s rate: $6000/2=3000\text{MHz}=3.0\text{GHz}$. So the balanced ratio is $\text{FCLK}:\text{MCLK}=3.0:3.0=1:1$ Insight: a 1:1 FCLK:MCLK ratio is exactly why enthusiasts tune FCLK to track memory speed — drop below it and the fabric starves the cores of the bandwidth the DIMMs can supply.

Test perfect squares directly with the exact mesh formula $\frac{2(N-1)}{3\sqrt{N}}$ vs ring $N/4$:

• $N=4$: mesh $=\frac{2\cdot 3}{3\cdot 2}=1.0$, ring $=1.0$ → tie, mesh not strictly less.
• $N=9$: mesh $=\frac{2\cdot 8}{3\cdot 3}=\frac{16}{9}\approx 1.78$, ring $=2.25$ → mesh strictly wins. $\boxed{N=9}\text{ is the smallest square where the mesh strictly beats the ring.}$ This matches the real-world "~10–12 core" crossover once you add routing and area overheads that push the practical break-even a little higher than the pure-math $N=9$. In short: the mathematics tips in the mesh's favour at $N=9$, but the messy engineering realities of building bigger routers delay the real switch until roughly a dozen cores.

Bus weakness ::: one transaction at a time + electrical loading (one shared wire). Ring average hops for N nodes ::: N/4 (exact). Exact mesh average hops for k×k grid ::: 2(N−1)/(3√N), which → 2√N/3 for large N. Why directory beats broadcast snooping ::: O(k) targeted probes vs O(N) broadcast, k = sharers. A read's worst-case latency counts how many fabric crossings ::: four (request out + data reply, each two crossings). Balanced FCLK:MCLK ratio ::: 1:1 so fabric bandwidth matches memory bandwidth. Smallest square N where mesh strictly beats ring ::: N = 9.