Infinity Fabric - mesh interconnects: Question bank

This is the "gotcha" companion to the parent topic. Each line below hides a trap — a place where a reasonable-sounding belief is actually wrong. Cover the answer, commit to a guess, then reveal. Every answer gives you the reason, not just a verdict.

Symbols and words used on this page

Before the traps, every symbol must be earned. These are the only ones this page uses:

Figure — Infinity Fabric - mesh interconnects

Look at the four pictures above. A bus (leftmost) is one shared black wire — everyone waits their turn. A ring joins nodes in a loop. A mesh is a grid where inner nodes have four neighbours. A torus (rightmost, red wrap-around links) is a mesh whose edges wrap around to the opposite side, so every node has four neighbours — no edge tiles. Keep this picture in mind for every question.

Two more mechanisms you must know before the traps

Some answers below lean on two hardware tricks. Earn them here first.

One-glance reminder of the prerequisite pages

You do not need to leave this page, but here is what each linked topic contributes:

Recall What the prerequisite links give you

6.3.1-Bus-ArchitectureBasics ::: why a single shared wire serializes traffic and hits an electrical-loading wall.
6.3.5-Cache-Coherency-Protocols ::: MOESI-style states and the difference between broadcast snooping and directory tracking.
6.2.8-NUMA-Architecture ::: why "near memory" and "far memory" have different latencies — the same effect chiplets create.
7.2.4-Network-Topologies ::: ring / mesh / torus / hop-count and bisection-bandwidth vocabulary.
6.4.3-PCIe-Topology and 5.1.7-Memory-BandwidthCalculation ::: where fabric bandwidth meets I/O and DRAM limits.

What "grows like" means — Big-O in one paragraph

A few answers compare how traffic grows as the machine gets bigger. We write this with Big-O notation: $O (N)$ means "the amount grows in direct proportion to $N$ " (double the cores, double the work), and $O (k)$ means "the amount grows in proportion to $k$ " and ignores everything that does not grow. It is a way to say how something scales, not an exact count. So "broadcast snooping is $O (N)$ " means its work climbs with the whole core count, while "a directory is $O (k)$ " means its work climbs only with the (small) sharer count.

Deriving the two hop-count formulas (so the recall lines are earned)

The traps below quote $N /4$ and $2 N /3$ . Here is why those averages arise, step by step, with a picture.

Formula Mesh average =

$2 N /3$ — with the gap derivation WHAT: on a $k \times k$ mesh, hops = horizontal distance $+$ vertical distance (the "Manhattan" walk — right then up, like city blocks). WHY split it: X and Y are independent, so we average each separately and add (red staircase in the figure). THE GAP DERIVATION (why $\frac{k}{3}$ ): put two tiles at random integer positions $i, j$ each drawn uniformly from $0 \dots k - 1$ . We want the average of $∣ i - j ∣$ . For a continuous position drawn uniformly on $[0, 1]$ the expected gap between two independent draws is a standard result: $E ∣ i - j ∣ = \int_{0}^{1} \int_{0}^{1} ∣ x - y ∣ d x d y = \frac{1}{3}$ Scaling that unit interval up to a line of length $k$ multiplies the gap by $k$ , giving $\frac{k}{3}$ per axis. Adding the two independent axes: $avg mesh hops = X \frac{k}{3} + Y \frac{k}{3} = \frac{2 k}{3} = \frac{2 N}{3}$

Counting mesh links properly (the trap correction, shown fully)

The "16 tiles ⇒ 16 links" error deserves its own picture, because the real count is not $2 N$ either — that formula ignores the missing edge links.

True or false — justify

A shared bus lets every core talk at the same time as long as the wire is fast enough

False. A bus is one shared medium — only one transaction is on it at a time no matter the clock; arbitration forces cores to take turns, so adding cores adds waiting, not parallel traffic.

Infinity Fabric is a single fixed topology baked into every AMD chip

False. It is topology-agnostic: the same protocol forms rings inside a chiplet, star-like spokes to a central I/O die, or full meshes between GPU dies depending on the product.

A mesh always beats a ring on latency for any core count

False. For a small number of cores (say 4–8) the ring's short circumference makes its average hops comparable, and the mesh's extra router ports cost area and power for little gain — that is why rings survived until ~10–12 cores.

A torus and a plain mesh have the same worst-case latency

False. A torus wraps its edges, so opposite corners are neighbours through the wrap link — its worst-case distance is roughly halved versus a plain mesh, at the cost of longer physical wires that are hard to route on a chip.

Cache coherency and data transport are the same job on the fabric

False. Transport (moving flits point-to-point) and coherency (deciding who may hold a line) are separate layers; the fabric carries coherency messages but the Scalable Coherent Protocol is the logic that decides what those messages are.

The Fabric Clock (FCLK) and the memory clock (MCLK) must always be equal

False. They run at a ratio — 1:1, 1:2, etc. A 1:1 ratio (coupled mode) avoids a synchronizer crossing and lowers latency; a 2:1 lets memory run faster than the fabric can keep pace with, at a latency cost.

A directory-based protocol sends a probe to every core on every write

False. That describes broadcast snooping. A directory tracks a sharer list per line and probes only the cores actually holding a copy — typically 1–3, not all

N

Point-to-point links mean data never has to be buffered

False. Packet-switched links use credit-based flow control and per-router buffers; a sender may only transmit a flit when it holds a credit for a free receiver slot, otherwise it stalls.

Bisection bandwidth measures how fast one link is

False. It measures how many links you must cut to split the network into two halves — a network-wide capacity, not a single-link speed. It reveals worst-case cross-chip traffic limits.

On EPYC Rome, a core reaching another chiplet's L3 is just as fast as reaching its own L3

False. Cross-chiplet access must hop out to the central I/O die and back, which is exactly why these designs expose NUMA-like distance effects (see 6.2.8-NUMA-Architecture).

More links in a mesh automatically means more usable bandwidth to any single destination

False. A single source-destination flow still uses one path at a time; the extra links help aggregate traffic and give alternate routes, not a wider single stream.

Spot the error

"Average ring hops are

N /2

, since a message might travel all the way around."

The error: you always take the shorter direction on a bidirectional ring, capping the trip at

N /2

and averaging to about

N /4

. Assuming

N /2

double-counts the long way you would never use.

"A 4×4 mesh has 16 links because it has 16 tiles."

The error: links and tiles are counted differently. Count by direction — 12 horizontal

+

12 vertical

= 24

links, i.e.

2 k (k - 1)

. The "

2 N = 32

" shortcut over-counts the edges that corner tiles are missing (see the link-count figure above).

"Directory coherency removes all snoop traffic, so it is free."

The error: it removes broadcast snoop traffic, not all of it. The home node still sends targeted probes to sharers and must be looked up on every access — cost scales with the sharer count (

k

= sharers), not zero.

"The I/O die is just a passive wiring hub with no logic."

The error: on Zen 2+ the I/O die hosts the memory controllers and the inter-chiplet fabric switching; it is an active routing and coherency participant, not passive copper.

"Because links are dedicated point-to-point, the fabric can never deadlock."

The error: dedicated links do not prevent cyclic buffer dependencies. Routers use multiple virtual channels specifically to break those cycles — see the deadlock-cycle figure below for exactly how.

"FSB (Front-Side Bus) bandwidth is low because the wire is thin; a wider bus fixes everything."

The error: the real limit is electrical loading — every attached core adds capacitance, capping the clock — plus single-transaction arbitration. Widening bits does not remove either problem.

"XY dimension-ordered routing sometimes takes the long way, so it wastes hops."

The error: XY routing always takes a minimal (Manhattan) path — X-distance plus Y-distance. It restricts which minimal path you pick (deadlock-free), never the length — see the routing figure below.

Visualising the two subtle mechanisms

Why a deadlock forms, and why virtual channels break it

Why XY routing never wastes hops, and which turns it forbids

Why questions

Why does a directory scale better than broadcast snooping as core count grows

Broadcast sends every request to every core, so traffic grows as

O (N)

(proportional to the whole core count). A directory probes only actual sharers, so traffic grows as

O (k)

where

k

= sharers (1–3) stays roughly constant regardless of

N

Why did Intel abandon the ring bus around Skylake-X

On a ring, latency grows with core count (average

N /4

hops) and the ring's shared bandwidth saturates past ~10–12 cores; a 2D mesh gives each node multiple shorter paths, keeping latency and bisection bandwidth in check.

Why is a 1:1 FCLK-to-MCLK ratio usually recommended for Ryzen desktop

A 1:1 (coupled) ratio removes an asynchronous clock-domain crossing between the fabric and memory, cutting the round-trip latency; running memory faster than the fabric (2:1) reintroduces that crossing penalty.

Why does the fabric bandwidth on a 5800X (≈57.6 GB/s one direction) happen to match dual-channel DDR4-3600

By design. If the fabric were slower than memory it would bottleneck memory reads; matching them (1:1 ratio) ensures the interconnect never becomes the limiting stage between DIMMs and cores (see 5.1.7-Memory-BandwidthCalculation).

Why is a "home node" needed at all in the coherency protocol

The home node (the L3 slice owning a line's address range) is the single serialization point that holds the directory, so all requests for that line route through one place and get a consistent order — preventing two cores from silently disagreeing about the line's state.

Why do chiplet designs accept extra cross-die latency instead of building one huge die

Smaller chiplets have far higher manufacturing yield and lower cost; a single 64-core monolithic die would be enormous and defect-prone. The latency penalty is the price paid for cheaper, more scalable silicon.

Why does packet-switching use flits rather than sending a whole cache line as one unit

Splitting into flits (flow-control units) lets flow control and buffering operate on small pieces, so a router needs only small buffers and can pipeline a message across many hops instead of storing it whole at each stop.

Edge cases

What is the average hop count on a mesh when source and destination happen to be the same tile

Zero hops — a core reaching its own local L2 slice never enters the router network. The

\frac{2 N}{3}

average includes this zero-distance case, which is exactly why the average is below the maximum.

On a 4×4 mesh, what is the worst-case (not average) hop count, and where does it occur

6 hops, from one corner

(0, 0)

to the opposite corner

(3, 3)

: a Manhattan distance of

3 + 3

. The average (

\approx 2.67

) hides this maximum, which matters for tail latency.

How does converting that 4×4 mesh into a torus change the worst case

The wrap-around links make

(0, 0)

and

(3, 3)

only

1 + 1 = 2

hops apart (one wrap step in X and one in Y), so the worst case drops from 6 to about 4 hops — the torus's edge-case advantage that a plain mesh cannot match.

What happens to a single-chiplet (monolithic) desktop Zen 4 part's inter-chiplet latency

There is none — with only one core die, all traffic stays on the internal fabric and never crosses an xGMI link, so the cross-chiplet NUMA penalty simply does not exist for that part.

If two cores both request the same line at the same instant, what breaks the tie

The home node serializes them: whichever request the home's directory logic orders first is handled first, and the second is treated as arriving afterward — there is never a true simultaneous grant.

What is the bisection bandwidth of a ring versus a mesh at the degenerate case of

N = 2

Both collapse: a 2-node "ring" is just one bidirectional link, and a

2 \times 2

mesh is not even well-defined. The mesh's advantage only appears once

N

is large enough to form a real grid.

If two cores both request the same line at the same instant, what breaks the tie

The home node serializes them: whichever request the home's directory logic orders first wins, and the second is treated as arriving afterward — there is never a true simultaneous grant.

What happens to a mesh's alternate-path advantage under strict XY routing

It gives zero benefit for a fixed source-destination pair — every such packet takes the same fixed path, so a hotspot on that path cannot be routed around even though other links sit idle.

Recall Quick self-test

$N$ and $k$ mean what ::: $N$ = number of nodes; $k$ = mesh side length so $k = N$ (and separately $k$ = number of sharers in coherency talk). Ring average hops formula and why ::: $N /4$ — average of the shorter-arc distances $1 \dots N /2$ , whose midpoint is $\approx N /4$ . Mesh average hops on a $k \times k$ grid and why ::: $\frac{2 k}{3} = \frac{2 N}{3}$ — expected gap $k /3$ per axis (from $\int \int ∣ x - y ∣ = \frac{1}{3}$ scaled by $k$ ), two axes added. Real link count of a 4×4 mesh ::: 24, from $2 k (k - 1)$ ; a torus reaches $2 N = 32$ by adding wrap links. What a credit is ::: a token meaning "one free receiver buffer slot"; the sender spends one per flit and stalls when out. Why virtual channels exist ::: separate buffer lanes on one wire that let blocked flits take an alternate lane, breaking deadlock cycles. Big-O meaning ::: shorthand for how a quantity scales — $O (N)$ grows with the whole node count, $O (k)$ only with the small sharer count.