6.2.12 · D5GPU Architecture
Question bank — Tensor cores and matrix operations
Before you start, we lean on a handful of words and symbols. We define every one before using it — including the floating-point formats and the software libraries the questions later assume.
Two more visuals you'll be pointed back to. Glance at them now:
Figure s03 — the warp→fragment mapping. Each of the 32 warp threads holds scattered cells of the output tile (colours show which thread owns which cell), so no thread owns a clean row — that is why the mapping is called "opaque."
Figure s04 — the roofline. Performance rises with arithmetic intensity until it hits the compute ceiling; left of the ridge you are memory-bound and the Tensor Cores sit idle, right of it they run at full speed.
True or false — justify
True or false: A Tensor Core makes a single thread multiply a 4×4 matrix.
False. The instruction is warp-level (the WMMA operation) — all 32 threads of the warp cooperate, each holding only a tiny fragment of A, B, and the accumulator (see figure s03). No lone thread ever sees a whole tile.
True or false: Tensor Cores replace CUDA cores.
False. They live alongside CUDA cores inside the same SM. CUDA cores still handle everything that isn't a dense matrix multiply — activations, reductions, control flow, FP64 on older chips.
True or false: Using FP16 inputs means the whole matrix multiply is done in FP16.
False. Inputs are FP16 but the running sum is kept in FP32 (the accumulator). That's the whole point of mixed precision — cheap multiplies, safe additions.
True or false: TF32 uses 32 bits of precision.
False. Despite the name, TF32 keeps only a 10-bit mantissa (like FP16) with FP32's exponent range. The "32" refers to the exponent range, not the total precision. Mantissa error is , not .
True or false: BF16 and FP16 are just different names for the same 16-bit float.
False. Both are 16 bits, but BF16 spends more bits on the exponent (wide range, coarse steps) and FP16 more on the mantissa (narrow range, fine steps). BF16 rarely overflows on gradients; FP16 does — see 8.3.4-Mixed-precision-training.
True or false: Tensor Cores speed up any matrix multiply you throw at them.
False. If the operation is memory-bound (see the roofline model, figure s04, and 9.1.5-Roofline-model), the cores sit idle waiting for data. They only help when you have enough arithmetic per byte to keep them fed, and dimensions that tile cleanly.
True or false: Because matrix multiply is associative, tiling and accumulating partial products gives exactly the FP32 result.
False in floating point. Associativity holds in exact math, but reordering additions changes rounding. Results are close, not bit-identical — a real source of surprise when comparing runs.
True or false: Sparse ("structured sparsity") Tensor Cores just skip zeros automatically for you.
False. On Ampere/Hopper the 2:4 structured sparsity feature needs weights pruned to a fixed pattern (2 nonzeros in every group of 4), then compressed offline. Only that enforced pattern earns the extra ~2× — arbitrary zeros give nothing.
True or false: Turning on Tensor Cores requires rewriting your model in the WMMA API.
False. Libraries like cuBLAS and cuDNN (and PyTorch on top of them) dispatch to Tensor Cores automatically when dtype and shapes qualify. The WMMA (warp-level MMA) API is only for hand-written kernels.
Spot the error
"I set my matrices to float16 but got no speedup, so Tensor Cores must be broken."
The likely error is shape. If , , aren't multiples of 8/16, the library falls back to CUDA cores. Check , , alignment before blaming hardware.
"I'll accumulate in FP16 too — it's faster and saves memory."
The error is accuracy. Over additions, small FP16 rounding errors compound and can swamp the result. The FP32 accumulator exists precisely to stop this drift; dropping it defeats mixed precision.
"My inference model uses INT8 weights, so the accumulator is also INT8."
Wrong. INT8 products are summed into an INT32 accumulator. Adding hundreds of INT8×INT8 products would overflow 8 bits almost immediately — see 8.4.2-Quantization-techniques.
"Volta Tensor Cores are physically 16×16, that's why the WMMA tile is 16×16."
The error conflates two levels. The hardware sub-partition is 4×4; the 16×16 is the WMMA API-exposed tile built from many 4×4 sub-multiplies across the warp. The programmer sees 16×16; the silicon works in 4×4.
"Each thread in the warp computes one element of the output tile."
False mapping. The warp→fragment mapping is opaque and non-obvious (see figure s03) — a thread holds scattered pieces, not one clean output cell. Never assume
threadIdx indexes an output element."I quantized to INT8 for training to make it faster."
The error is using INT8 for training. INT8 is an inference technique after training; gradients need the range/precision of BF16 or FP16. INT8 lacks the dynamic range for backprop.
"cuBLAS is slow, so I'll write my own WMMA kernel — it'll beat it."
Usually wrong. cuBLAS is hand-tuned for tiling, memory staging, and pipelining across the memory hierarchy. Naive WMMA kernels typically lose badly to it.
Why questions
Why not just make the Tensor Core tile 64×64 instead of 4×4 — bigger is faster, right?
A tile must fit in the warp's registers. Too large and it spills to slower memory (a net loss), and it wastes hardware on edge cases where matrices don't fill it. 4×4 balances register fit against amortizing instruction-decode cost.
Why is the accumulator wider than the inputs at all?
Because a sum of products spans a much larger magnitude range than any single product (see figure s02). A wide FP32/INT32 accumulator captures that range without overflow or losing small terms to rounding.
Why do dimensions being multiples of 8 matter, when the API tile is 16×16?
Two different granularities collide. The 16 comes from the WMMA API tile size; the 8 comes from how a warp's fragments pack into the register file (and, for some precisions, the -step is 8 not 16). A dimension that isn't a multiple of the smaller granularity, 8, already breaks clean fragment packing and drops you to a slow path — so 8, not 16, is the real floor for "will this hit Tensor Cores at all."
Why does TF32 give "free" speedup with no code changes?
Because it accepts ordinary FP32 tensors and silently truncates the mantissa inside the Tensor Core. Your code still declares FP32; the hardware does the trimming, so the API surface is unchanged (Ampere+ default).
Why do Tensor Cores exist if CUDA cores could do the same math?
A CUDA core does one FMA per clock; deep learning needs billions. A Tensor Core packs 64 FMAs into one instruction, cutting instruction count 64× and boosting throughput 8–16× — see 6.2.1-GPU-vs-CPU-architecture.
Why can a Tensor Core be idle even during a big matrix multiply?
If data can't arrive fast enough (bandwidth-bound region of the roofline, figure s04), the cores starve. Peak FLOPS only appear when arithmetic intensity is high enough to hide memory latency — see 9.1.5-Roofline-model.
Edge cases
What happens to a 17×17 matrix multiply on 16×16 tiles?
The library must round each dimension up to the next multiple of 16: in both dimensions, so a padded region covers four 16×16 tiles. Useful work is cells out of , so is wasted padding — a tiny overshoot costs as much as the next tile boundary up.
What if (the inner dimension) is 1 — a vector outer product?
There's almost no arithmetic to amortize the tile's overhead, and no accumulation loop to reuse loaded data. It's strongly memory-bound; Tensor Cores give little to no benefit here.
What if you feed FP32 tensors on a Volta GPU (no TF32)?
Volta Tensor Cores accept only FP16 inputs, so pure FP32 matmul runs on CUDA cores, not Tensor Cores. You get no Tensor Core speedup at all until you cast to FP16.
What if a weight is exactly zero after INT8 quantization?
It multiplies cleanly to zero and contributes nothing to the INT32 sum — no special handling. Quantization's real risk is nonzero weights snapping to the wrong integer, not the zeros.
What happens on FP64 scientific work on an early Tensor Core?
Early (Volta/Turing) Tensor Cores have no FP64 mode, so the work runs on CUDA cores. Only Ampere/Hopper add FP64 Tensor Core paths for range-sensitive simulations.
What if a matrix contains an FP16 value that overflows during multiply?
FP16's narrow range (max ) can overflow to before accumulation even happens. This is why gradient-heavy work prefers BF16 or loss-scaling — see 8.3.4-Mixed-precision-training.
What about underflow — a gradient too small for FP16?
Below FP16 slides into subnormals (reduced precision) and then flushes to zero — the gradient silently vanishes. Loss-scaling multiplies the loss up before backprop to lift these values back into FP16's range.
What happens if one input is a NaN (e.g. upstream)?
NaN propagates: any FMA touching it produces NaN, which then poisons the whole accumulator for that output cell. One bad value can turn an entire output tile into NaN, so guard against and at the source.
What does structured sparsity require to actually run faster on Ampere/Hopper?
The weights must be pruned to the 2:4 pattern (exactly 2 nonzeros per group of 4) and compressed offline; the hardware then skips the known zeros for ~2× more throughput. Random sparsity gives zero benefit — see 8.4.2-Quantization-techniques.
Recall Quick self-test
One trap you got wrong above ::: Re-read its justification and phrase the underlying rule in your own words before moving on. The single most common myth here ::: "Tensor Cores do everything in low precision" — no, inputs are low precision, accumulation is high precision. What do M, N, K stand for? ::: = output rows, = output columns, = the shared inner dimension summed over. What is the difference between MMA and WMMA? ::: MMA is the matrix multiply-accumulate operation itself; WMMA is that same operation issued cooperatively by a whole 32-thread warp through NVIDIA's API.