Worked examples — Infinity Fabric - mesh interconnects

This is the "worked examples" deep-dive for the parent topic on Infinity Fabric & mesh interconnects. The parent showed you what the fabric is and gave two starter examples. Here we hunt down every case class — every shape of question the exam or a real design review can throw at you — and work each one from zero.

Before any number appears, remember the three quantities we keep using, in plain words:

Two clock names appear again and again below; meet them now so no symbol is ever a surprise:

One more term used repeatedly in the bandwidth examples:

---

The scenario matrix

Every question about this topic falls into one of these cells. Our job is to leave no cell untouched.

Cell	What makes it different	Covered by
A. Bandwidth — one direction	frequency × width, single lane	Ex 1
B. Bandwidth — bidirectional / ratio	double it, then check the FCLK:MCLK ratio	Ex 2
C. Routing — generic hop count	Manhattan distance on a mesh	Ex 3
D. Routing — degenerate (same row / same node)	zero in one axis, or zero total	Ex 4
E. Ring vs mesh — the crossover	when does mesh actually win? sign of the difference	Ex 5
F. Bisection bandwidth — the "cut"	count links crossing a slice	Ex 6
G. Chiplet latency — the multi-hop word problem	mix ns and cycles across clock domains	Ex 7
H. Coherency traffic — O(N) vs O(k)	limiting/scaling behaviour of probes	Ex 8
I. Exam twist — mismatched ratio bottleneck	fabric slower than memory: who wins?	Ex 9

We will hit A→I in order.

---

Cell A — Bandwidth, one direction

Forecast: guess the order of magnitude first — tens of GB/s? Hundreds? Write your guess down.

1. Convert frequency to ticks per second. $1.8\text{ GHz}=1.8\times 10^9$ cycles/s. Why this step? Bandwidth is bytes per second, so we need seconds in the denominator. The GHz already gives us "per second."
2. Multiply width by ticks. Each tick moves $32$ bytes, so $B{W}_{\text{1-dir}}=32\frac{\text{B}}{\text{cycle}}\times 1.8\times 10^9\frac{\text{cycle}}{\text{s}}=57.6\times 10^9\frac{\text{B}}{\text{s}}.$ Why this step? Units cancel: cycle over cycle disappears, leaving bytes/second — exactly what "bandwidth" means.
3. Name it. $57.6\times 10^9\text{ B/s}=57.6\text{ GB/s}$. Why this step? $10^9\text{ B/s}$ is by definition one gigabyte per second (GB/s), so we rewrite the number in the unit everyone quotes in datasheets — same value, human-readable name.

Verify: Units: $\frac{\text{B}}{\text{cycle}}\cdot \frac{\text{cycle}}{\text{s}}=\frac{\text{B}}{\text{s}}$ ✓. Sanity: 32 bytes is half a 64-byte cache line, so almost 2 billion half-lines move per second — comfortably in the "tens of GB/s" range.

---

Cell B — Bidirectional & the clock ratio

Forecast: does bidirectional simply double? And is the fabric geared $1:1$ or $1:2$ against this memory?

1. Double it. A link has a separate wire for each direction, so $B{W}_{\text{bi}}=2\times 57.6=115.2\text{ GB/s}.$ Why this step? Send and receive run at the same time on independent lanes — they don't share, so we add them.
2. Find MCLK from the DDR label. "DDR4-3600" means $3600\text{ MT/s}$ (mega-transfers/s). DDR is double data rate — two transfers per clock — so the memory clock is $\text{MCLK}=\frac{3600}{2}=1800\text{ MHz}=1.8\text{ GHz}$. Why this step? The ratio compares clock to clock, so we must strip the "double" out of the transfer rate to recover the actual MCLK.
3. State the true ratio. $\text{FCLK}=1.8\text{ GHz}$ and $\text{MCLK}=1.8\text{ GHz}$, so the gearing is $1:1$ — not $2:1$. They tick in lockstep. Why this step? Naming the ratio correctly is the whole point of Cell B; a $1:1$ fabric moves one packet per memory tick, the ideal AMD tuning target.
4. Compute the memory bandwidth (one direction). This platform has two channels (dual-channel — see the Memory channel definition above), each $8\text{ B}$ wide, at $3.6\times 10^9$ transfers/s: $B{W}_{\text{mem}}=2\text{ch}\times 3.6\times 10^9\times 8=57.6\text{ GB/s (one direction)}.$ Why this step? We must compare fabric to memory in the same direction to see if the fabric can keep up with a read stream — and the channel count is read off the platform, not assumed.

Verify: $2\times 57.6=115.2$ ✓; $\frac{3600}{2}=1800\text{ MHz}$ so ratio $=1:1$ ✓. Fabric-per-direction ($57.6$) $=$ memory-per-direction ($57.6$), so a read burst drains into the fabric without piling up. Cell I below shows what happens when this ratio is broken.

---

Cell C — Generic routing hop count

Forecast: is it the straight-line distance, or the grid-walk distance?

Figure 1 — XY routing on a 4×4 mesh. Alt: a grid of 16 dots; a teal source dot at (1,0) and a plum destination at (3,3) joined by an orange arrowed path that goes east 2 steps then north 3 steps, turning exactly once. This picture makes the point that the packet walks the grid, never diagonally.

1. Define the distance we need. On a grid you cannot fly diagonally; you walk along links. The right measure is the Manhattan distance:
2. X first (the "X" of XY routing). From $x=1$ to $x=3$: $|3-1|=2$ hops east. Why X first? Dimension-ordered routing fixes an order (X, then Y) so packets never chase each other in a loop — this prevents deadlock (parent note, virtual channels).
3. Then Y. From $y=0$ to $y=3$: $|3-0|=3$ hops north.
4. Add. Total $=2+3=5$ hops. Path: $(1,0)\to (2,0)\to (3,0)\to (3,1)\to (3,2)\to (3,3)$ — look at the orange path in Figure 1: it turns exactly once.

Verify: Manhattan $=|3-1|+|3-0|=2+3=5$ ✓. A single turn confirms XY routing (X leg, then Y leg).

---

Cell D — Degenerate routing (zeros)

Before the degenerate cases, let us build the average-hop formula from scratch so this page stands on its own.

Picture picking two nodes at random on one axis of a line that is $\sqrt{N}$ dots long. On average the two picks land about a third of the way apart — the expected gap is $\frac{\sqrt{N}}{3}$. (Intuition: if you drop two points uniformly on a stick of length $L$, the average distance between them is $\frac{L}{3}$ — the points are more often near each other than at opposite ends.) A mesh has an X axis and a Y axis, and the two gaps simply add:

$$\text{avg hops}=\underset{\text{X gap}}{\underbrace{\frac{\sqrt{N}}{3}}}+\underset{\text{Y gap}}{\underbrace{\frac{\sqrt{N}}{3}}}=\frac{2\sqrt{N}}{3}.$$

Keep in mind: this is an average over all pairs, not the distance of any single specific route. That distinction is exactly what the degenerate cases below expose.

Forecast: what does "average hops $=\frac{2\sqrt{N}}{3}$" give here, and is it wrong?

1. Case (a): same X column. $|2-2|+|3-1|=0+2=2$ hops (purely Y). Why call it out? The averaged formula $\frac{2\sqrt{N}}{3}\approx 2.67$ would predict $\approx 2.67$ — but that is an average over all pairs, not this specific pair. For one aligned pair the X term is exactly zero. Never apply the average to a single deterministic route.
2. Case (b): source = destination. $|2-2|+|2-2|=0$ hops. The message never leaves the local port. Why this matters: zero hops means zero router-to-router latency; the only cost is the local agent handoff. Degenerate but real (a core hitting its own L2 slice).

Verify: (a) $0+2=2$ ✓; (b) $0+0=0$ ✓. Both are $\le$ the maximum possible hop count $2\times (4-1)=6$, and $\ge 0$ — inside valid bounds.

---

Cell E — Ring vs mesh crossover (sign of the win)

First, the two symbols this example lives on — do not read past them until they are clear:

Forecast: at tiny $N$ (say 4 cores), which is lower? Guess before solving.

Figure 2 — hop-count crossover. Alt: two curves on a warm-paper plot; a teal straight line N/4 (ring) and an orange curve 2·sqrt(N)/3 (mesh) crossing near N=7.1, marked with a plum dashed line; left of the crossing the ring is lower, right of it the mesh is lower. The picture shows the exact core count where mesh starts to win.

1. Set up the inequality. Mesh wins when its hop count is smaller: $\frac{2\sqrt{N}}{3}<\frac{N}{4}.$ Why this step? $T_{\text{hop}}$ is the same for both, so it cancels — the winner depends purely on hop count.
2. Solve for $N$. Multiply both sides by 12: $8\sqrt{N}<3N$, i.e. $8<3\sqrt{N}$, so $\sqrt{N}>\frac{8}{3}\approx 2.67$, giving $N>{\left(\frac{8}{3}\right)}^2=\frac{64}{9}\approx 7.1$. Why algebra and not just plugging numbers? We want the crossover, not one data point — solving the inequality gives the boundary directly.
3. Round to a real mesh. The smallest perfect square with $N>7.1$ is $N=9$ ($3\times 3$).
4. Check the sign at the boundary. At $N=4$: ring $=1.0$, mesh $\approx 1.33$ hops → ring wins (mesh loses at tiny core counts!). At $N=16$: ring $=4$, mesh $\approx 2.67$ → mesh wins. The sign of $(\frac{N}{4}-\frac{2\sqrt{N}}{3})$ flips between them.

Verify: $N=9$: mesh $\frac{2\cdot 3}{3}=2.0$, ring $\frac{9}{4}=2.25$ → mesh smaller ✓. $N=4$: mesh $1.333>$ ring $1.0$ ✓ (ring wins, matching the parent's "~10–12 core" real-world crossover once router overhead is added). See also 7.2.4-Network-Topologies for the general topology trade-off.

---

Cell F — Bisection bandwidth (the cut)

Forecast: which topology has more wires spanning the middle?

Figure 3 — the bisecting cut. Alt: two side-by-side diagrams; left, a 16-node ring with a teal dashed horizontal cut severing exactly 2 links; right, a 4×4 mesh with a teal dashed vertical cut severing 4 links (one column). The mesh clearly has twice as many links crossing. This is why mesh bisection bandwidth scales with √N.

1. Define bisection bandwidth. Cut the network into two equal halves; bisection bandwidth is the total bandwidth of every link severed by that cut. Why this metric? If the two halves talk to each other at full tilt, this is the ceiling on the traffic between them — the real "how much can flow across" number.
2. Ring cut. A ring is a single loop; slicing it in half severs exactly 2 links (the two arcs). See the teal dashed cut in Figure 3.
3. Mesh cut. A $4\times 4$ mesh cut down the middle severs one column of vertical links: 4 links ($\sqrt{N}=4$). Why $\sqrt{N}$? The cut runs across a full row/column of the grid, and a $\sqrt{N}\times \sqrt{N}$ grid has $\sqrt{N}$ links spanning it.
4. To bandwidth. Ring $=2\times 115.2=230.4\text{ GB/s}$; mesh $=4\times 115.2=460.8\text{ GB/s}$ → mesh has $2\times$ the bisection bandwidth.

Verify: ratio $\frac{4}{2}=2$ ✓; $4\times 115.2=460.8$ ✓; $2\times 115.2=230.4$ ✓. This is why mesh scales bandwidth with $\sqrt{N}$ while ring stays flat.

---

Cell G — Chiplet latency word problem (mixed units)

Forecast: which leg dominates — the on-chip cycles or the die-crossing ns?

1. Convert cycles to ns. One cycle at $3.5\text{ GHz}$ lasts $\frac{1}{3.5\times 10^9}\text{ s}=0.2857\text{ ns}$. So $5$ cycles $=5\times 0.2857=1.43\text{ ns}$. Why convert first? You cannot add cycles and nanoseconds directly — they live in different clock domains. Bring everything to one unit (ns).
2. Add the die crossings. $20+20=40\text{ ns}$ for the two fabric hops. Why just add them? Latencies along a single serial path accumulate — the packet does one leg, then the next, so total time is the sum.
3. Total. $1.43+40=41.43\text{ ns}$. Why this is the answer? Everything is now in nanoseconds, so the three serial legs add to the end-to-end latency: $41.43\text{ns}$.

Verify: $\frac{5}{3.5}=1.4286\text{ ns}$ ✓; total $=1.4286+40\approx 41.43\text{ ns}$ ✓. The $40\text{ ns}$ of die-crossing dwarfs the $1.4\text{ ns}$ on-chip — this is the "chiplet penalty" the parent mentions, and why 6.2.8-NUMA-Architecture matters: distant memory feels farther.

---

Cell H — Coherency traffic scaling (limiting behaviour)

Forecast: does directory traffic grow with total cores or just with sharers?

1. Broadcast: probe everyone. Every write sends a probe to all other cores: $N-1$ messages. At $N=8$: $7$; at $N=64$: $63$. Why $N-1$? Broadcast has no knowledge of who has the line, so it must ask all cores except the writer — this is the $O(N)$ scaling the parent warns about.
2. Directory: probe only sharers. The home L3 slice knows the sharer list, so it sends $k$ probes regardless of $N$. At $N=8$: $3$; at $N=64$: $3$. Why constant? Directory tracks sharers explicitly, so traffic is $O(k)$ — flat as the chip grows. This is the whole reason mesh CPUs use directories (see 6.3.5-Cache-Coherency-Protocols).
3. Ratio at $N=64$. Broadcast/directory $=\frac{63}{3}=21\times$ fewer messages with a directory. Why take the ratio? It expresses the saving in a size-free way — "21 times less traffic" is the headline a designer cares about, independent of units.

Verify: $N-1$: $7$ and $63$ ✓; directory: $3$ and $3$ ✓; ratio $\frac{63}{3}=21$ ✓. Broadcast blows up linearly; directory stays flat — the limiting behaviour that makes 64-core fabrics feasible.

---

Cell I — Exam twist: broken ratio bottleneck

Forecast: with faster memory, do you get faster real bandwidth — or does something upstream cap you?

1. Fabric one-direction bandwidth. $32\times 2.0\times 10^9=64\text{ GB/s}$. Why one direction? A read stream flows memory→core in a single direction; compare like with like.
2. Memory one-direction bandwidth. Given as $96\text{ GB/s}$ (two channels, $8\text{ B}$ each, $6000\text{ MT/s}$).
3. The bottleneck is the minimum. Data must pass through both; the slower stage caps the throughput: $B{W}_{\text{eff}}=\min (64,96)=64\text{ GB/s}.$ Why min and not average? A pipeline runs at the speed of its slowest stage — the fabric is a hard wall the memory bytes must squeeze through.
4. The lesson. Faster RAM with a lagging FCLK wastes $96-64=32\text{ GB/s}$ of memory bandwidth. The fix: raise FCLK (or accept the $1:2$ penalty). Compare with 6.4.3-PCIe-Topology where a slow root link similarly caps device throughput.

Verify: fabric $32\times 2.0=64$ ✓; $\min (64,96)=64$ ✓; wasted $96-64=32$ ✓. The exam trap: bigger memory number does not mean bigger real bandwidth.

---

Recall Quick self-test

A $4\times 4$ mesh, $(0,1)\to (2,1)$, XY routing — how many hops? ::: $|2-0|+|1-1|=2$ hops (Cell D, X-only) Fabric $32\text{ B}$ at $1.8\text{ GHz}$, bidirectional? ::: $2\times 32\times 1.8=115.2\text{ GB/s}$ (Cell B) Directory probes for $k=3$ sharers on a 64-core chip? ::: $3$, independent of N (Cell H) Ring vs mesh at $N=4$ — which wins on hops? ::: Ring ($1.0<1.33$) — mesh loses at tiny N (Cell E) Fabric $64\text{ GB/s}$, memory $96\text{ GB/s}$ — effective? ::: $\min =64\text{ GB/s}$ (Cell I)