Question bank — CUDA cores and execution model

CUDA core ::: A single arithmetic-logic unit (ALU) that does one FP32 or integer op per clock — not a full CPU-style core. Warp ::: A fixed group of 32 threads that execute the same instruction in lockstep under one shared scheduler. SM (Streaming Multiprocessor) ::: The real "processor" unit — a bundle of 64–128 CUDA cores plus schedulers, registers, and shared memory. SIMT ::: Single Instruction, Multiple Thread — one instruction drives many threads, but each thread keeps its own register state and can diverge. Occupancy ::: How many warps are resident on an SM relative to the maximum — the fuel supply for latency hiding.

False. ::: A CUDA core is just an ALU — it has no instruction decoder, no branch predictor, and no private cache. It shares all control logic with 31 other cores in its warp, so it cannot run an independent instruction stream the way a CPU core can.

False. ::: Cores are yoked together in warps of 32 under shared control, so you get at most (cores ÷ 32) independent instruction streams, and even those are grouped per SM. Independence lives at the warp level, not the core level.

False. ::: A warp instruction is atomic per issue — its minimum latency is one cycle. Extra cores give the SM throughput (it can issue several warps' instructions the same cycle), not lower latency for any single warp.

False. ::: 32 is a fixed hardware convention — a power of two that simplifies address masking and thread indexing, inherited from the 8/16/32 SIMD lineage. SMs with 64 or 128 cores still use 32-thread warps; they just issue more warps per cycle.

cannot take different branches. False. ::: They can diverge — each thread owns its register state. But when they do, the warp serialises: it runs the if path with the else-threads masked off, then the else path with the if-threads masked off. Divergence is legal but slow, not forbidden.

False. ::: Only up to the point where the latency window is fully covered ($T_{\text{required}}=\frac{S\cdot L}{C}$). Beyond that, extra warps sit idle — they add nothing because the ALUs were already kept busy. See occupancy.

True. ::: A block is guaranteed to reside on a single SM for its whole lifetime — that is why its threads can share fast shared memory and synchronise with barriers. The grid, by contrast, spreads across all SMs.

same address. False. ::: Coalescing means threads read consecutive addresses (thread 0 → A[0], thread 1 → A[1], …) so the hardware fuses them into one 128-byte transaction. All-reading-the-same address is a broadcast, a different (also efficient) pattern.

$\eta$ is measured in gigabytes per second. False. ::: $\eta =\frac{\text{useful bytes}}{\text{transactions}\times 128}$ is dimensionless — bytes over bytes, a fraction between 0 and 1. Effective bandwidth is $\eta$ times peak DRAM bandwidth; $\eta$ itself is just the wastage ratio.

False. ::: In SIMD one instruction fills fixed vector lanes with no per-lane control flow. In SIMT each thread has its own program counter and register state and may diverge — the hardware masks lanes to fake independence. SIMT is SIMD with a per-thread illusion of freedom.

$S=4$ schedulers and $C=10$ compute cycles per warp, I need $\frac{200}{10}=20$ warps." Error: dropped the scheduler count. ::: $T_{\text{required}}=\frac{S\cdot L}{C}=\frac{4\times 200}{10}=80$ warps. Each cycle the SM can issue $S$ warp-instructions, so the latency window has capacity $S\cdot L$, not just $L$.

$T_{\text{required}}=80$, so I'll just raise occupancy to 80 warps." Error: 64 is a hard ceiling. ::: You cannot exceed the SM's resident-warp cap. Since $80>64$, full hiding is impossible by adding warps — you must instead raise compute intensity $C$ (do more work per memory op) to lower $T_{\text{required}}$.

Error: transactions multiply. ::: A stride-32 pattern turns one 128-byte transaction into up to 32 transactions for the same useful bytes, so $\eta$ falls from $100\%$ to $3.125\%$ — up to 32× slower, not 2×.

Error: block cap is 1024. ::: A single block holds at most 1024 threads (it must fit on one SM). Massive parallelism comes from launching many blocks in the grid, not from oversized blocks.

Error: zero-overhead switching. ::: All resident warps have their registers already loaded in the SM's register file. Switching warps is free — the scheduler just picks a different ready warp next cycle. This is the whole point of keeping many warps resident.

sin(x) in one cycle just like a multiply." Error: wrong unit. ::: CUDA cores handle FP32 FMA, integer, and logic ops. Transcendentals like sin/exp run on separate special-function units (SFUs) on the SM, at lower throughput — not on the CUDA cores.

::: Because a warp shares one instruction stream. When threads diverge, the hardware runs each path sequentially with the other threads masked off — an if/else split can halve throughput, and deep nesting compounds it. GPUs win only when the same instruction applies to all 32.

::: Blocks give a scalability abstraction: 1000 blocks run on a 10-SM GPU (100 blocks/SM) or an 80-SM GPU (~12 blocks/SM) with no code change. They also define the scope of shared memory and barrier synchronisation. See parallel patterns.

$C$ help more than adding warps once you hit the warp cap? ::: Because $T_{\text{required}}=\frac{S\cdot L}{C}$ shrinks as $C$ grows. If you can't add warps (cap reached), lowering the requirement is the only lever left — do more arithmetic per memory fetch so fewer warps are needed to cover the same stall.

::: 32 is a power of two, which makes thread-index computation and lane-masking a matter of cheap bit operations (masking, shifting) rather than division. Non-powers-of-two would complicate every address-generation step in hardware.

::: The SM has a fixed register file (~65,536 32-bit registers on many recent SMs). If each thread grabs many registers, fewer threads fit: roughly $\frac{M}{P}$ registers per thread for $P$ resident threads. Register-hungry kernels therefore lower occupancy and weaken latency hiding.

specifically at the warp level? ::: Because the hardware inspects the 32 addresses a warp requests together and tries to fuse them into minimal 128-byte transactions. Coalescing is a property of the warp's collective access pattern, not of any single thread's access.

if (i < N). What do the other 12 do? ::: They are masked off — they still occupy their lanes and consume the cycle, but their results are discarded. The warp cannot run faster than 32-wide; the extra 12 lanes are simply wasted work at the array tail.

N is not a multiple of the block size. Is that a bug? ::: No — the standard idiom numBlocks = (N + threadsPerBlock - 1) / threadsPerBlock rounds up, launching a few extra threads, and the if (i < N) guard masks the overhang. The final warp is partly masked, which is normal.

::: $\lceil 100/32\rceil =4$ warps (96 + 4 threads). The fourth warp runs with only 4 active lanes and 28 masked-off — wasted capacity. This is why block sizes are usually chosen as multiples of 32.

A[0]. Coalesced or catastrophic? ::: Neither wasteful — this is a broadcast. The hardware fetches the 128-byte line once and hands the value to all 32 threads, so $\eta$ stays high. Uniform reads are efficient; it's scattered reads that hurt.

different code paths. Worst-case slowdown? ::: Up to 32× — the warp must serialise all 32 paths one at a time, running each with a single active lane. This is the pathological limit of branch divergence.

::: There is nothing to switch to. The moment that warp stalls on memory (100–400 cycles), the SM's ALUs sit idle the whole time — no hiding at all. This is why single-warp kernels waste the GPU almost completely, a key lesson for training kernels.

::: No. FP64 runs on separate, far fewer double-precision units on the SM. On consumer GPUs FP64 throughput can be a small fraction (e.g. 1/32) of FP32 — so a DP-heavy kernel bottlenecks even with cores "available."