Intuition The one core ideaInside a modern CPU there are dozens of little "workers" (cores, caches, memory controllers) that constantly need to send messages to each other, and the wiring between them is a tiny road network. Infinity Fabric and mesh interconnects are that road network — the whole topic is about choosing road shapes (bus, ring, grid) so that messages arrive fast without all traffic jamming on one road.
This page assumes you know nothing . Every letter, symbol, and buzzword the parent note throws at you is built here, in order, with a picture for each. Read top to bottom.
Before we can draw roads, we need to know who is driving on them.
A node is any single "box" on the chip that can send or receive messages: a CPU core, a slice of cache, a memory controller, or an I/O controller. In every picture on this page a node is drawn as a coloured square .
A core is one full CPU brain — it runs your program instructions. A chip with "8 cores" has 8 of these, and they all want data at the same time. That "at the same time" is the entire reason we need a clever network.
A cache is a small, very fast memory that sits next to cores and keeps recently-used data close. A big shared cache is chopped into pieces called slices ; each slice lives next to a node. Caches matter because two cores can hold copies of the same data — and copies can disagree (we fix that in section 8).
Building block for prerequisites: if the words "core", "cache", and "memory controller" feel fuzzy, the wiring background in 6.3.1-Bus-ArchitectureBasics is a good companion — this page builds directly on it. (We define every wiring word we use, including bus , in section 2 below, so you can keep reading right here.)
Look at the figure: three squares (a core , a cache slice , a memory controller ) each with a short stub — that stub is the plug-in point where a road will later attach. (We give this plug-in point its proper name, port , in section 3.) Keep this cast of characters in mind; every topology below just rearranges these same squares.
Now we connect the squares. There are three shapes the parent note keeps comparing.
A bus is one shared wire that every node hangs off. Only one node may "talk" at a time — everyone else waits. Picture a single narrow corridor where people must take turns.
A ring connects nodes in a circle : each node wires only to its two neighbours. A message to a far node must hop past every node in between. Picture a circular hallway — to reach the room opposite you, you walk halfway around.
A mesh is a 2D grid of nodes. Each node wires to its neighbours up/down/left/right (North, South, East, West). Picture city blocks: many routes exist from any corner to any other corner.
Intuition Why the shape mattersThe bus (left) jams because everyone shares one wire. The ring (middle) is fair but slow for far trips . The mesh (right) gives shortcuts. The parent note's whole "why fabric replaced rings and buses" story is just: as we add cores, the bus jams and the ring gets slow, so we switch to the mesh.
Topology is simply "the shape of the network" — the pattern of which node connects to which. Bus, ring, and mesh are three topologies. See 7.2.4-Network-Topologies for the same idea at data-center scale.
To talk about "fast" and "slow" we need to count.
A hop is one step of a message from one node to the next-door node. A trip across many nodes is measured in number of hops . In a hallway, one hop = walking past one doorway.
Definition Router (= switch)At each grid intersection sits a router : a tiny traffic cop that reads where a message wants to go and pushes it out the correct exit. In Intel's mesh each router has 5 ports .
⚠️ You will also see the word switch . In this topic "router" and "switch" mean the exact same component — the little traffic cop at a tile. We use "router" throughout; just read "switch" as a synonym whenever the parent note uses it.
A port is one doorway on a router. Intel's mesh router has 5: North, South, East, West, and one local port going down to the node it sits on (its own core + cache slice). (This is the "plug-in point" you saw as a stub back in figure s01.)
The figure shows a single 5-port router. The four grey arrows are the compass directions to neighbour routers; the magenta arrow going down is the local port to this router's own core. Every mesh in the parent note is just many of these tiles wired edge-to-edge.
A tile = one core + its cache slice + its router, packaged as a repeatable unit. Build a chip by stamping tiles into a grid. That is literally what "mesh interconnect" means physically.
The parent note writes positions like ( 0 , 0 ) ( 3 , 2 )
Definition Grid coordinates( x , y ) x y ( 0 , 0 ) need coordinates so we can compute distances with arithmetic instead of eyeballing.
Why this formula and not straight-line distance? On a grid you cannot cut diagonally through buildings — you must travel along streets, East/West then North/South. So the honest hop count adds the horizontal and vertical legs. That is why it's named after Manhattan's block layout.
Worked example Reading the parent's routing exampleFrom ( 0 , 0 ) ( 3 , 2 ) ∣3 − 0∣ + ∣2 − 0∣ = 3 + 2 = 5 5 hops .
Common mistake Don't forget the barsWriting x 2 − x 1 ∣ ⋅ ∣ − 3
N N how many nodes (usually cores) are on the chip. The parent uses N = 16 N = 64 N any chip size.
N k N square grid, each side has N N k = N . k N = 16 k = 16 = 4 4 × 4 k N
N k = N perfect squares (16 , 25 , 36 , 64 …
Rectangular meshes. Use two side-lengths, k x k y k x ⋅ k y = N 6 × 3 = 18 ≈ k x /3 ≈ k y /3 ≈ ( k x + k y ) /3 k x = k y = N Rounding a "square" estimate. When someone quotes N N approximation ; pick the nearest rectangle ⌊ N ⌋ × ⌈ N / ⌊ N ⌋⌉ ⌊ ⋅ ⌋ ⌈ ⋅ ⌉
Bottom line: treat k = N square special case; for messy core counts swap in k x , k y
The parent note writes E [ ∣ x 2 − x 1 ∣ ]
E [ ⋅ ] E [ something ] average value of something over all the random cases". Read it as "typical" .
Definition The uniform-traffic assumptionTo get one clean number we assume uniform random traffic : every source tile is equally likely, and every destination tile is equally likely, independently. This is the standard "no favourites" baseline. Real chips can have hot spots (e.g. everyone hammering one memory controller) — we handle that edge case at the end of this section.
Deriving E [ ∣ x 2 − x 1 ∣ ] = 3 k k 2 − 1 We look only at the x x 1 x 2 0 , 1 , … , k − 1 k k 2 k 2
E [ ∣ x 2 − x 1 ∣ ] = k 2 1 S a = 0 ∑ k − 1 b = 0 ∑ k − 1 ∣ a − b ∣ . Now we actually compute the sum S , not just quote it. Group the pairs by their gap g = ∣ a − b ∣
gap g = 0 ( 0 , 0 ) , ( 1 , 1 ) , … k 0
gap g = 1 ( 0 , 1 ) , ( 1 , 0 ) , ( 1 , 2 ) , ( 2 , 1 ) , … k − 1 2 orderings , so 2 ( k − 1 ) 1
gap g k − g 2 2 ( k − g ) g
Add them up:
S = g = 1 ∑ k − 1 g ⋅ 2 ( k − g ) = 2 g = 1 ∑ k − 1 ( k g − g 2 ) . Use the two schoolbook sums ∑ g = 1 k − 1 g = 2 ( k − 1 ) k ∑ g = 1 k − 1 g 2 = 6 ( k − 1 ) k ( 2 k − 1 )
S = 2 [ k ⋅ 2 ( k − 1 ) k − 6 ( k − 1 ) k ( 2 k − 1 ) ] = 3 ( k − 1 ) k ( k + 1 ) = 3 k ( k 2 − 1 ) . Divide by k 2
E [ ∣ x 2 − x 1 ∣ ] = k 2 S = k 2 k ( k 2 − 1 ) /3 = 3 k k 2 − 1 ≈ 3 k . For large k − 1 3 k The useful 3 1 ∑ g 2
The row part is identical (swap x y 3 k
avg hops = 3 k + 3 k = 3 2 k = 3 2 N .
Recall Where does the ring's4 1 N N n shorter arc, so the true distance is min ( n , N − n ) N avg = N 1 ∑ n = 0 N − 1 min ( n , N − n ) . N n = 0 … 2 N is n n = 2 N … N − 1 0 2 N ∑ n = 0 N − 1 min ( n , N − n ) = 2 ∑ n = 1 N /2 − 1 n + 2 N = 2 ⋅ 2 ( N /2 − 1 ) ( N /2 ) + 2 N = 4 N 2 . ∑ n = 1 m n = 2 m ( m + 1 ) m = 2 N − 1 2 N N avg hops (ring) = N 1 ⋅ 4 N 2 = 4 N . 4 1 ∑ min ( n , N − n ) = N 2 /4 N 0 N /2 N /4
Common mistake Uniform is amodel , not a law
If traffic is non-uniform — say every core repeatedly hits one "home" memory controller — the k /3 N /4
The parent's worked examples multiply bytes by frequency. Here is every piece.
Definition Clock frequency (GHz)A clock is a metronome that ticks billions of times per second. 1 GHz = 1 billion ticks (cycles) per second. On each tick, hardware moves one batch of data.
Definition FCLK — Fabric ClockFCLK is the clock that drives Infinity Fabric specifically. A higher FCLK means more data batches per second across the fabric. (MCLK is the memory clock; the parent talks about their 1:1 or 2:1 ratio.)
Definition Bus width (bytes per cycle)The width is how many bytes the road carries per tick . A "32 B wide" fabric moves 32 bytes each cycle. Wider road = more per tick.
Worked example Reproducing the parent's Example 1One direction: 32 × 1.8 = 57.6 2 × 57.6 = 115.2
Why multiply by 2? The link has separate wires for send and receive, so both directions run simultaneously — you get double the total.
For the full memory-side of this arithmetic, see 5.1.7-Memory-BandwidthCalculation .
Definition Bisection bandwidthCut the network cleanly into two halves and count the roads that cross the cut. Bisection bandwidth = how much data can flow between the two halves at once. It measures the worst-case chokepoint .
Ring: any straight cut severs exactly 2 links (the circle is crossed in two places).k × k k = N row — that is k = N
So the general rule is: ring bisection = 2 links, mesh bisection = N For N = 16 16 = 4 2 × grows with N
Definition Cache coherenceWhen two cores each keep a copy of the same data, coherence is the rule-system that keeps those copies in agreement — so nobody reads stale data after someone else writes. Without it, one core could "correct" a value that another core never sees change.
Definition Directory vs broadcast
Broadcast snoop : shout the update to every core. Traffic grows with N Directory-based : a bookkeeper (the "home" cache slice) records exactly which cores hold a copy, and messages only those few. Traffic grows with the number of sharers k s k s k
Infinity Fabric uses the directory idea so coherence stays cheap even with 64 cores. Deep dive: 6.3.5-Cache-Coherency-Protocols .
The home node for a piece of data is the fixed cache slice responsible for tracking it. Every request for that data routes through its home first — like every parcel for an address going through one local post office.
A chiplet is a small separate silicon die holding a cluster of cores. AMD builds big CPUs by wiring several chiplets together with Infinity Fabric instead of making one giant die (cheaper, better yield).
Definition NUMA (Non-Uniform Memory Access)Because chiplets sit at different distances, reaching nearby memory is fast and far memory is slower — access time is non-uniform . That is the 6.2.8-NUMA-Architecture idea, and it's why the parent note carefully counts the "worst-case chiplet 0 → chiplet 7" hop cost.
Definition xGMI / IFIS linkThe named wire that carries Infinity Fabric between dies or packages. xGMI stands for inter-Chip Global Memory Interconnect (the "x" = inter/cross, "GMI" = Global Memory Interconnect; the on-package version is plain GMI ). IFIS stands for Infinity Fabric Inter-Socket — the same fabric extended across a socket-to-socket link between two physical CPU packages. Think of both as the highway bridge between two separate city grids: the tiles inside each die talk over the local mesh, and xGMI/IFIS carries traffic across the gap. This is also how external devices like PCIe endpoints reach into the chip's fabric.
Intuition Why irregular meshes existA silicon die is a rectangle, and the memory controllers and I/O land at fixed spots along its edges — not on a tidy grid. So real "meshes" are often irregular : a mostly-rectangular grid with a few tiles missing, extra links to reach an edge-mounted controller, or non-uniform row/column counts to match the die shape.
Definition Irregular / non-rectangular meshA mesh whose tiles do not form a perfect k × k k x × k y pattern of which router wires to which differs.
Common mistake The clean formulas are best-case guides3 2 N N perfect square mesh with uniform traffic. On an irregular die-shaped mesh, real average hops and bisection must be measured on the actual layout — the formulas give a ballpark , not a guarantee. Designers use them to compare topology choices , then simulate the exact chip.
coordinates and Manhattan distance
averages E of hops k over 3
bisection bandwidth sqrt N
cache coherence directory home
rectangular and irregular meshes
Infinity Fabric and mesh interconnects
Read it as: the raw players become tiles , tiles arrange into a topology , we measure that topology with hops and coordinates , size it with N k = N k x , k y E bandwidth , keep its copies honest with coherence , and finally accept real dies use irregular grids — all of which combine into the parent topic.
Cover the right side; can you answer each before moving to the parent note?
What is a node in one sentence? Any box on the chip (core, cache slice, memory/I-O controller) that sends or receives messages.
Why does a bus jam but a mesh not? A bus is one shared wire so only one node talks at a time; a mesh gives many parallel roads and shortcuts.
What is a hop ? One step of a message from a node to its immediate neighbour.
Do "router" and "switch" mean different things here? No — in this topic they are the same component, the 5-port traffic cop at a tile.
How many ports does an Intel mesh router have, and what are they? 5 — North, South, East, West, and one local port to its own core/cache.
Write the Manhattan distance from ( x 1 , y 1 ) ( x 2 , y 2 ) d = ∣ x 2 − x 1 ∣ + ∣ y 2 − y 1 ∣
For N k What if N Use a rectangular mesh k x × k y k x k y = N ≈ ( k x + k y ) /3
What does E [ ⋅ ] The average (typical) value over all random cases.
Why is the average column gap ( k 2 − 1 ) / ( 3 k ) ≈ k /3 Grouping pairs by gap g S = ∑ g 2 ( k − g ) g = k ( k 2 − 1 ) /3 k 2 ( k 2 − 1 ) / ( 3 k )
Where does the ring's N /4 ∑ n min ( n , N − n ) = N 2 /4 N N /4
What assumption makes k /3 N /4 Uniform random traffic (no favourites); hot-spot / non-uniform traffic breaks it.
Give the bandwidth formula. Bandwidth = bus width (bytes) × clock frequency (cycles per second); double it for bidirectional.
Compute one-direction bandwidth of a 32 B fabric at 1.8 GHz. 32 × 1.8 = 57.6
What is the bisection width of a N × N Mesh
= N links, ring
= 2 links.
Directory vs broadcast coherence — which scales better and why? Directory, because it messages only the few sharers (O ( k s ) O ( N )
What does xGMI stand for, and IFIS? xGMI = inter-Chip Global Memory Interconnect (die-to-die); IFIS = Infinity Fabric Inter-Socket (package-to-package).
Why do real meshes end up irregular? A die is a rectangle with I/O and memory controllers fixed at the edges, so the grid gets missing tiles and extra links to match the die shape.
Ready? Head back to the parent topic — every symbol there is now yours.
