Tensor cores and matrix operations: Visual walkthrough

This page builds the central idea of Tensor cores and matrix operations from absolutely nothing. We start with "what does it even mean to multiply two grids of numbers?" and end with "why does dedicated hardware make this 64× denser." Every step has a picture. Follow the coloured arrows.

If a word here feels assumed — it isn't. We build it.

Step 1 — What is a matrix, really?

WHAT. A matrix is just a rectangular grid of numbers. We give it a name (a capital letter like $A$ ) and we address any single number inside it with two labels: which row it's in and which column it's in.

WHY. Before we can "multiply" grids, we must agree on how to point at one number inside a grid. That pointing is the whole game.

PICTURE. Look at the figure. The number in row $i$ , column $j$ is written $A [i] [j]$ . The little square that lights up is exactly that entry.

Figure — Tensor cores and matrix operations

A grid that is $M$ rows tall and $K$ columns wide is called an " $M \times K$ " grid. Read the " $\times$ " as "by": "M by K". That is size, not multiplication yet.

Step 2 — What does "multiply two grids" mean? (one output cell)

WHAT. To multiply grid $A$ (size $M \times K$ ) by grid $B$ (size $K \times N$ ), we produce a new grid $C$ (size $M \times N$ ). We compute one cell of $C$ at a time.

WHY. A single rule, applied to every output cell, defines the whole operation. So we only ever need to understand one cell. Master that, master everything.

PICTURE. To get $C [i] [j]$ (the highlighted cell of the output), we grab row $i$ of $A$ (a horizontal strip) and column $j$ of $B$ (a vertical strip). We slide along both together, multiply the pair we land on, and keep a running total.

Step 3 — Why is this the expensive part? (counting the work)

WHAT. First, a name for the basic operation. Each "multiply one pair, add it to a running total" is called a fused multiply-add (FMA) — a multiply and an add glued into one operation. Now we count how many FMAs the whole grid costs.

WHY. Deep learning is mostly this operation. If we know the FMA cost, we know why anyone bothered to build special hardware.

PICTURE. Each output cell needs $K$ FMAs (one per slide position). There are $M \times N$ output cells. So the tower of work is $M \times N \times K$ tall.

Recall

Why does matrix multiply cost $M \cdot N \cdot K$ ? ::: $M \cdot N$ output cells, each needing $K$ multiply-adds to fill.

This is the wall. A CUDA core does one FMA per clock. A billion FMAs, one per clock, is a lot of clocks. Enter the tile.

Step 4 — The trick: chop the grids into 4×4 tiles

WHAT. Instead of one giant multiply, cut $A$ , $B$ , $C$ into little $4 \times 4$ blocks called tiles. The big multiply becomes many small tile-multiplies that we add up.

WHY. The single dot-product sum for a cell (Step 2) can be split into groups of 4 slide positions and the group-totals added back together — because addition is associative (regrouping a sum of products never changes the total). Each group of 4 is exactly a $4 \times 4$ tile-multiply, so we can hand each tile to a dedicated hardware unit and add up the results.

PICTURE. One output tile of $C$ (top-left $4 \times 4$ block) is built by walking a strip of $A$ -tiles across and a strip of $B$ -tiles down, multiplying each opposite pair of tiles, and accumulating into the same output tile.

Step 5 — One tile = one instruction (what a Tensor Core is)

WHAT. A Tensor Core is hardware that computes an entire $4 \times 4$ tile multiply-accumulate, $D = A \times B + C$ , in one instruction — not 64 separate steps.

WHY. That $4 \times 4$ tile hides $4 \times 4 \times 4 = 64$ FMAs. Doing all 64 in one shot is the whole point: you pay the instruction-decode cost once for 64 arithmetic operations.

PICTURE. Left: a CUDA core grinding through 64 tiny FMAs one clock at a time. Right: a Tensor Core swallowing all 64 in a single instruction cycle. Same answer, 64× fewer instructions.

Step 6 — The scoreboard: how much did tiling save?

WHAT. Count Tensor Core instructions for the $102 4^{3}$ job and compare to the scalar FMA count.

WHY. This is the payoff line — the number that justifies the whole chapter.

PICTURE. Two towers side by side: 1.07 billion scalar FMAs vs 16.8 million tile-instructions. The ratio is a clean $64$ .

Whether this 64× turns into 64× wall-clock speedup depends on feeding the cores data fast enough — that's the memory hierarchy and roofline story. Compute density is necessary, not sufficient.

Step 7 — The degenerate cases (never leave the reader stranded)

WHAT. Handle the awkward inputs: a dimension not divisible by 4, $K = 0$ , and a genuinely empty output ( $M = 0$ or $N = 0$ ).

WHY. Real matrices aren't always neat multiples. The reader must know what happens at every edge.

PICTURE. A grid whose width isn't a multiple of 4: the last tile hangs off the edge. We pad the missing cells with zeros so the tile is still $4 \times 4$ .

The one-picture summary

Everything above, on one canvas: index a cell → dot a row with a column → count the mountain of FMAs → chop into $4 \times 4$ tiles → one tile = one Tensor Core instruction → 64× fewer instructions.

Recall Feynman retelling — say it back in plain words

A matrix is a grid of numbers. To multiply two grids, each output number is made by sliding along one row of the first grid and one column of the second, multiplying the numbers you land on together and adding them all up. Each multiply-and-add is one FMA. Do that for every output cell and you've paid $M \times N \times K$ FMAs — for 1024-sized grids that's over a billion, way too many to do one at a time.

The clever move: cut the grids into little $4 \times 4$ blocks. Because you can add a big pile of products in bundles and get the same total as adding them in a line (addition is associative), you can multiply block-by-block and add the block-answers up. A Tensor Core is a chunk of silicon that eats one whole $4 \times 4$ block-multiply-and-add in a single instruction — 64 multiply-adds for the price of one instruction. Across the whole $102 4^{3}$ job that's exactly 64× fewer instructions.

Edge cases don't break it: if a grid's size isn't a multiple of 4, you pad the gap with zeros (zeros add nothing, so the answer is unchanged — you just waste a bit of the instruction). If the shared dimension is zero, the sum is empty and the accumulator $C$ just passes through. If a whole dimension is zero, the output has no cells and there's simply nothing to do. That's the whole idea: same arithmetic as ordinary matrix multiply, packed into fixed-size tiles so dedicated hardware can blast through it.

See also: mixed-precision training and quantization use these tiles at FP16/BF16/INT8; the Hinglish note covers the same ground.