Worked examples — Tensor cores and matrix operations

This page is the drill ground for Tensor Cores. The parent note told you what a Tensor Core is and why it exists. Here we count out every kind of situation you can be handed — every awkward matrix shape, every precision choice, every "does it even fit?" edge case — and work each one to the number.

If you have not yet met the machinery of tiled matrix multiply, skim the parent first. Everything else we build from zero right here.

---

Two pieces of notation, built first

Two symbols recur on every line below. We define them before using them.

---

The scenario matrix

Before working problems, let us enumerate the cases. A Tensor Core problem is fully described by a few knobs. If a knob can take a value that breaks the naive approach, that value gets its own row.

Cell	Case class	What is being tested	Example
A	Clean fit	Dimensions are exact multiples of the tile	Ex 1
B	Ragged fit (needs padding)	Dimension not a multiple of tile → padding / performance cliff	Ex 2
C	Degenerate: $K=0$, and $M=0$ / $N=0$	The sum / the output has zero size — what is the answer?	Ex 3
D	Skinny / vector-like ($N=1$ or $M=1$)	Tile mostly wasted — utilisation question	Ex 4
E	Instruction-count / speedup arithmetic	Counting FMAs and Tensor Core ops	Ex 5
F	Precision choice (FP16 vs BF16 vs TF32 vs INT8)	Which mode, and the accuracy cost	Ex 6
G	Accumulator overflow / range	When low precision silently fails	Ex 7
H	Real-world word problem (roofline / timing)	Bytes vs FLOPs, is it compute-bound?	Ex 8
I	Exam-style twist (transpose / layout)	row_major vs col_major, off-by-tile traps	Ex 9

We will use these symbols throughout — defined once, here:

---

Ex 1 — Cell A · The clean fit

What we are doing: counting how many tile-sized bites cover the whole output.

Figure below (s01). Alt-text: a blue $256\times 128$ output rectangle is sliced by a thin gray grid into $16$ tiles down and $16$ tiles across; one tile is filled orange and labelled "one $16\times 8$ output tile ($D$)"; a green arrow labelled "$K/k=32$ accumulation steps" points into that orange tile to show it is built up by summing $32$ K-chunks. Read the three grid counts straight off the picture.

1. Count output-row tiles. $M/m=256/16=16$ (the $16$ rows of the grid in s01). Why this step? The output is $256\times 128$; along its height we lay down tiles of height $m=16$. It divides evenly → Cell A, clean.
2. Count output-column tiles. $N/n=128/8=16$ (the $16$ columns of the grid). Why? Same logic across the width, tile width $n=8$.
3. Count inner (K) tiles. $K/k=512/16=32$ (the green arrow's depth). Why? Each output tile is not finished in one shot — the shared dimension $K$ is sliced into chunks of $k=16$, and we accumulate those chunks. Each chunk's product is added onto the running tile via the $+C$ in $D=A\times B+C$: the previous step's $D$ is fed back as this step's $C$.
4. Multiply the three counts. $16\times 16\times 32=8192$ instructions. Why multiply? Every output tile (there are $16\times 16$ orange squares' worth) needs one mma per K-chunk (there are $32$).

Verify: total FMAs the hard way $=M\cdot N\cdot K=256\cdot 128\cdot 512=16,777,216$. Each mma does $m\cdot n\cdot k=16\cdot 8\cdot 16=2048$ FMAs. Then $16,777,216/2048=8192$. ✓ Same number two ways.

---

Ex 2 — Cell B · The ragged fit (padding & the performance cliff)

What we are doing: handling a dimension the tile size does not divide.

Figure below (s02). Alt-text: a horizontal strip of $17$ tiles; the first $16$ tiles ($128$ columns) are solid blue and labelled "real, clean"; the $17$th tile is split — an orange slice ($2$ columns, "real: 129–130") next to a gray hatched slice ($6$ columns, "padding: 131–136, wasted"). A red bracket over the $17$th tile reads "one extra tile for 2 extra columns = the cliff." Watch how the tiny orange sliver drags in a whole gray tile.

1. Test divisibility. $130/8=16.25$ — not an integer. This is the ragged case. Why this step? Tensor Cores only speak in whole tiles. A partial column strip cannot be issued as "16.25 tiles."
2. Round up with the ceiling. Number of column-tiles $=\lceil 130/8\rceil =17$ (recall $\lceil 16.25\rceil =17$, defined above) — the $17$ tiles in s02. Why round up, not down? Dropping the extra $2$ columns would lose data. We must cover columns 129 and 130 (the orange sliver), so we allocate a full $17$th tile.
3. Compute padded width. $17\times 8=136$ columns of tile space for $130$ real columns. Why? The last tile spans columns 129–136; columns 131–136 are padding (zeros in $B$, garbage-then-discarded in $D$) — the gray hatched slice.
4. Wasted columns $=136-130=6$. Why care? Those $6$ columns still cost full Tensor Core cycles. This is the performance cliff: adding 2 real columns cost you a whole extra tile.

Verify: wasted fraction of the last tile $=6/8=75\%$ idle in that strip. Overall extra work vs. a clean $128$: instructions go from $16\cdot 16\cdot 32$ to $16\cdot 17\cdot 32=8704$, i.e. $8704/8192=1.0625$, a 6.25% overhead to serve 2 extra columns. ✓

---

Ex 3 — Cell C · Degenerate sizes ($K=0$, and $M=0$ or $N=0$)

What we are doing: evaluating the tile counts and the defining sum at every boundary size.

Figure below (s03). Alt-text: three small panels side by side. Panel (a) "$K=0$": a solid blue $16\times 16$ output square exists but is filled with the label "$0$" — the output tiles are real, the contraction is empty. Panel (b) "$M=0$": a flat blue line of zero height labelled "$0\times 128$ — no rows, no tiles." Panel (c) "$N=0$": a thin blue vertical line of zero width labelled "$256\times 0$ — no cols, no tiles." Under all three a gray caption reads "instructions run $=0$ in every case."

1. Case (a), $K=0$ (empty contraction). The defining sum $C[i][j]={\sum }_{k=0}^{K-1}A[i][k]B[k][j]$ has an empty index range. By the universal convention a sum over zero terms is the additive identity $0$, so $C$ is the $16\times 16$ zero matrix (panel (a): a real square, filled with $0$). Why 0 and not "undefined"? An empty sum must leave an accumulator unchanged; the identity that "adds nothing" is $0$. Instruction count: $\lceil M/m\rceil \cdot \lceil N/n\rceil \cdot \lceil K/k\rceil =1\cdot 2\cdot 0=0$ mma calls. The output tiles exist ($16\times 16$ is non-empty) but there is nothing to accumulate.
2. Case (b), $M=0$ (no output rows). $C$ is $0\times 128$ — an empty matrix (no elements at all; panel (b) is a zero-height line). Why? If $A$ has zero rows, the product has zero rows; there is literally no $C[i][j]$ to compute. Instruction count: $\lceil 0/16\rceil \cdot \lceil 128/8\rceil \cdot \lceil 512/16\rceil =0\cdot 16\cdot 32=0$.
3. Case (c), $N=0$ (no output columns). $C$ is $256\times 0$ — again an empty matrix (panel (c) is a zero-width line). Why? Symmetric to (b): zero output columns → nothing to write. Instruction count: $\lceil 256/16\rceil \cdot 0\cdot \lceil 512/16\rceil =0$.
4. The unifying rule. If any of the three tile counts is $0$, the product runs zero instructions. But the reason differs: in (a) the output tiles exist and must still be initialised to zero; in (b)/(c) there are no output tiles to initialise at all. Why mention initialisation? Cell C's classic bug: for $K=0$ you must still fill_fragment(c_frag, 0.0f) or you return garbage — even though no multiply ever happens.

Verify: (a) $C$ is $16\times 16$ zeros, $1\cdot 2\cdot 0=0$ ops; (b) $C$ is $0\times 128$, $0$ ops; (c) $C$ is $256\times 0$, $0$ ops. ✓

---

Ex 4 — Cell D · Skinny matrix (matrix–vector, $N=1$)

What we are doing: measuring waste when a dimension is far smaller than the tile.

Figure below (s04). Alt-text: one $16\times 8$ output tile drawn as an $8$-wide, $16$-tall blue grid. The leftmost column is filled orange and labelled "useful output (1 column of $y$)"; the remaining $7$ columns are gray with diagonal hatching, labelled "wasted tile width (7 of 8 columns padding)." A red caption underneath reads "utilisation $=1/8=12.5\%$." Watch the orange strip occupy exactly one-eighth of the blue tile.

1. Output shape. $y$ is $4096\times 1$. Column-tiles needed $=\lceil 1/8\rceil =1$. Why? Even a single column forces a full width-$8$ tile — one orange column, but a whole blue tile allocated (see figure).
2. Useful columns per tile $=1$ out of $n=8$ (the orange strip vs. the whole blue tile in s04). Why? Columns 2–8 of that tile compute against padding (zero) columns of $x$ — real cycles, discarded results (the gray hatched region).
3. Column utilisation $=1/8=12.5\%$. Why this matters? Tensor Cores shine on fat GEMMs. A matrix–vector product wastes $87.5\%$ of the width dimension — it is memory-bound, not compute-bound (see Ex 8).
4. The row dimension is fine. $M=4096$ is a multiple of $16$, so height utilisation is $100\%$. Overall tile utilisation $=100\%\times 12.5\%=12.5\%$. Why separate the dimensions? Utilisation multiplies across independent tile axes.

Verify: column util $=1/8=0.125$; overall $=1\cdot 0.125=0.125=12.5\%$. In s04, the orange strip (real output) is exactly one of the eight columns of the blue tile. ✓

---

Ex 5 — Cell E · Speedup arithmetic

What we are doing: dividing total work by work-per-instruction.

Figure below (s05). Alt-text: a horizontal bar chart with a logarithmic x-axis (instruction count). A long orange bar reaches $\approx 1.07\times 10^9$ labelled "scalar FMAs (one per instruction)"; a much shorter blue bar reaches $\approx 5.24\times 10^5$ labelled "Tensor Core mma ops (2048 FMAs each)." A red annotation between them reads "$\times 2048$ fewer instructions." The blue bar is visibly a tiny fraction of the orange.

1. Total FMAs. $M\cdot N\cdot K=1024^3=1,073,741,824\approx 1.07\times 10^9$ (the orange bar). Why cube? Each of the $M\cdot N$ outputs needs $K$ FMAs.
2. FMAs per mma. $m\cdot n\cdot k=16\cdot 8\cdot 16=2048$. Why? One instruction fills an $m\times n$ output tile using a length-$k$ contraction $=mnk$ FMAs.
3. Reduction factor. $1024^3/2048=524,288$ instructions (the blue bar), i.e. the instruction count drops by a factor $2048$. Why the tile product? Every FMA hidden inside one instruction is one fewer instruction you issue.

Verify: $1024^3=1073741824$; $/2048=524288$ Tensor Core ops; factor $=2048$. ✓

Recall Why doesn't 2048× fewer instructions mean 2048× faster?

Because raw speedup is bounded by clock, memory bandwidth, and pipeline fill — instruction count is only the ceiling. Real speedup is 8–16×, set by hardware throughput, not by the FMA-per-op ratio. ::: Instruction reduction is an upper bound; memory and clock cap the actual gain.

---

Ex 6 — Cell F · Choosing the precision (FP16 vs BF16 vs TF32 vs INT8)

What we are doing: exercising all four precision modes on concrete numbers.

Figure below (s06). Alt-text: two stacked panels. Top panel — three horizontal bit-layout bars (FP32, BF16, FP16) of equal total length, each split into a blue "sign" cell, an orange "exponent" segment, and a green "mantissa" segment; FP32 and BF16 share the same wide orange exponent (labelled "range: 8 exp bits"), while FP16's exponent is visibly shorter ("range: 5 exp bits") and its mantissa longer than BF16's. A red arrow marks "$6\times 10^{-8}$ falls off FP16's range." Bottom panel — a number line from $-1$ to $1$ with $9$ evenly spaced orange ticks illustrating INT8's coarse grid, annotated "step $=2/256=0.0078$."

What we are doing: exercising all four precision modes on concrete numbers.

1. (i) FP16 vs BF16 — the range test. Compare $6\times 10^{-8}$ to each format's smallest normal. $6\times 10^{-8}<6.1\times 10^{-5}$ → FP16 underflows it toward zero. $6\times 10^{-8}\gg 1.2\times 10^{-38}$ → BF16 keeps it. Why? FP16 spends only 5 bits on the exponent (narrow range, top panel of s06); BF16 keeps FP32's full 8-bit exponent, trading mantissa bits for range. Choose BF16 for training.
2. (ii) TF32 — the free speed test. Relative rounding error is about one unit in the last mantissa bit: $2^{-10}\approx 9.77\times 10^{-4}$, versus FP32's $2^{-23}\approx 1.19\times 10^{-7}$. Why care? $\approx 10^{-3}$ relative error is under $0.1\%$ — invisible to most models — and TF32 needs no code change while running $\sim 8\times$ faster than FP32. Choose TF32 as the default fast FP32.
3. (iii) INT8 — the quantisation step test. With $256$ levels ($2^8$) spread across a range of width $1-(-1)=2$, the step between neighbouring representable weights is $2/256=0.0078125$ (the tick spacing, bottom panel of s06). Why this matters? Any weight is rounded to the nearest multiple of $\approx 0.0078$; INT8 accepts this coarse grid in exchange for $16\times$ throughput and $4\times$ smaller weights vs FP32 — fine for inference after training. Choose INT8 for post-training inference.

Verify: (i) $6\times 10^{-8}<6.1\times 10^{-5}$ and $>1.2\times 10^{-38}$; (ii) $2^{-10}=0.0009765625$, $2^{-23}\approx 1.19\times 10^{-7}$, and $2^{-10}<10^{-3}$; (iii) $2/256=0.0078125$. ✓ See 8.3.4-Mixed-precision-training and 8.4.2-Quantization-techniques.

---

Ex 7 — Cell G · Accumulator overflow (why FP32 accumulate exists)

What we are doing: bounding the accumulator and comparing to its capacity.

1. Worst-case per-product magnitude. $127\times 127=16129$. Why max? $127$ is the largest signed INT8 value; the worst single term.
2. Worst-case sum. $K\times 16129=4096\times 16129=66,064,384$. Why multiply by $K$? The contraction adds $K$ such terms; the worst case is all at max sign.
3. Compare to INT16 ceiling. $66,064,384>32767$ → massive overflow after just a handful of terms. Why does this ruin everything? An overflowed INT16 wraps around silently → garbage output, no error thrown.
4. Compare to INT32 ceiling ($2,147,483,647$). $66,064,384<2.1\times 10^9$ → fits comfortably. Why hardware uses INT32 accumulate: it guarantees no overflow for realistic $K$, which is exactly why Tensor Cores pair INT8 inputs with an INT32 accumulator (and FP16 inputs with FP32) — the accumulate result $D$ lives in the wide type even when $A,B$ are narrow.

Verify: $127^2\cdot 4096=66064384$; $>32767$ (INT16 overflows); $<2147483647$ (INT32 safe). ✓ This is the concrete reason "mixed precision" always accumulates wide.

---

Ex 8 — Cell H · Real-world word problem (compute- vs memory-bound)

What we are doing: applying the roofline model — the single test that says "compute or memory."

1. FLOPs. Matrix–vector does $M\cdot K$ FMAs $=4096\cdot 4096=16,777,216$ FMAs $=2\times$ that in FLOPs $=33,554,432\approx 3.36\times 10^7$ FLOPs. Why $\times 2$? Each FMA is a multiply and an add.
2. Bytes moved. We must read $W$ (dominant): $4096^2$ FP16 elements $\times 2$ bytes $=33,554,432$ bytes $\approx 3.36\times 10^7$ B. ($x$ and $y$ are negligible.) Why $W$ dominates? $x$ is only $4096$ elements; $W$ is $4096^2$ — a million times more data.
3. Arithmetic intensity $=\text{FLOPs}/\text{bytes}=33,554,432/33,554,432=1.0$ FLOP/byte. Why this ratio? It tells us how much compute we squeeze from each byte fetched.
4. Machine balance $=312\times 10^{12}/2000\times 10^9=156$ FLOP/byte. Compare: our intensity $1.0\ll 156$ → memory-bound by a huge margin.
5. Time estimate (memory). $t=\text{bytes}/\text{BW}=33,554,432/(2000\times 10^9)\approx 1.68\times 10^{-5}$ s $=16.8\mu$s. Why use bandwidth, not FLOP/s? When memory-bound, the memory pipe sets the time; the Tensor Cores idle.

Verify: intensity $=1.0$; balance $=156$; $1.0<156$ (memory-bound); $t=3.3554432\times 10^7/2\times 10^{12}=1.6777\times 10^{-5}$ s. ✓ (Compute-only time would be $3.36\times 10^7/3.12\times 10^{14}\approx 0.1\mu$s — far shorter, confirming memory dominates.)

---

Ex 9 — Cell I · The exam twist (layout / transpose trap)

What we are doing: tracking layout flags as transpose operators, because that is exactly what they are.

1. What layout means. Reading a row-major buffer as if col-major is the same as reading the transpose. So the col_major load flag applies a transpose $(\cdot {)}^{\top }$ to whatever bytes it reads. Why? Row-major $B$ stored, read column-major, yields $B^{\top }$ mathematically.
2. Student pre-transposes $B$ in memory → the buffer now holds $B^{\top }$. Why they did it: to "get" the transpose they wanted.
3. Then the col_major flag transposes again → the fragment holds $(B^{\top }{)}^{\top }=B$. Why this is the trap: two transposes cancel. $(X^{\top }{)}^{\top }=X$.
4. Result: mma_sync computes $A\times B$ — not the desired $A\times B^{\top }$. The two flips cancelled; the student silently gets the wrong product (no error). Fix: do exactly one transpose — either flip the buffer or set the layout flag, never both.

Verify: with $A=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]$, $B=\left[\begin{array}{cc}5& 6\\ 7& 8\end{array}\right]$: intended $A{B}^{\top }=\left[\begin{array}{cc}17& 23\\ 39& 53\end{array}\right]$ but double-flip gives $AB=\left[\begin{array}{cc}19& 22\\ 43& 50\end{array}\right]$. The two matrices differ → confirms the bug. ✓ (See 7.3.6-cuBLAS-and-cuDNN: libraries expose an explicit transA/transB flag so you flip once, deliberately.)

---

Recap

Recall Which cell forces you to round the tile count

up with the ceiling? Cell B (ragged fit): a dimension not divisible by the tile size uses $\lceil D/t\rceil$ tiles, creating padding and a performance cliff. ::: The ragged / padding case — use ceiling, expect wasted columns.

Recall Empty sizes: how do

$K=0$, $M=0$, and $N=0$ differ? All three run 0 mma instructions. But $K=0$ still produces a real (zero-filled) output tile that must be initialised, whereas $M=0$ or $N=0$ produce an empty matrix with no output tiles at all. ::: Zero instructions in every case; only $K=0$ still needs accumulator init.

Recall Why does INT8 Tensor Core hardware accumulate in INT32?

Because a length-$K$ contraction of INT8 products can reach $\sim K\cdot 127^2$, which overflows INT16 for even modest $K$ but stays well inside INT32's $\pm 2.1\times 10^9$ range. ::: To avoid silent overflow of the accumulated sum.

Recall Skinny matmul (

$N=1$): compute-bound or memory-bound? Memory-bound — arithmetic intensity $\approx 1$ FLOP/byte sits far below the machine balance ($\sim 156$), and tile utilisation is only $12.5\%$. ::: Memory-bound and low utilisation; batch to fix.

Related depth: 6.2.1-GPU-vs-CPU-architecture, 6.2.8-Memory-hierarchy-in-GPUs, 6.2.10-Warp-scheduling-and-execution.