Intuition The one idea behind the whole topicA GPU can do maths thousands of times faster than it can fetch a number from its main memory, so the entire game is keeping the numbers you need close to the arithmetic units . The memory hierarchy is just a ladder of storage boxes — tiny-and-instant at the top, huge-and-slow at the bottom — and every performance trick in this chapter is about which box a piece of data lives in.
Before you can read a single formula in the parent note, you need the vocabulary those formulas are built from. This page lists every symbol and every concept the parent assumes, from absolute zero, and orders them so each one rests on the one before.
Everything in this topic hangs on one image: storage arranged as a ladder . High rungs are small but instant to reach; low rungs are enormous but slow. Look at the figure below and hold it in your head — every later idea attaches to a rung.
Definition Latency and capacity — the two axes of the ladder
Latency = how long you wait for one piece of data to arrive, measured in clock cycles . Picture it as the length of the arrow you must walk to reach a rung.Capacity = how much fits on that rung, measured in bytes. Picture it as the width of the rung.
The whole hierarchy exists because these two fight each other: fast storage is built from expensive transistors, so it must be small.
Why the topic needs these two words: every row of the parent's summary table is nothing but a (latency, capacity) pair. If you don't feel these as distance and width , the table is just numbers.
A clock cycle is one tick of the GPU's internal metronome — the smallest slice of time in which the hardware does one basic step. All latencies in this topic are counted in ticks, not seconds, because ticks are what the hardware actually experiences.
Intuition Why count in cycles, not nanoseconds?A GPU running at 1.5 GHz ticks 1.5 billion times per second, so one tick ≈ 0.67 nanoseconds. But if the clock speeds up, every latency in nanoseconds shrinks together. Counting in cycles keeps the numbers stable and comparable across GPUs — that's why the parent says "registers: 1 cycle, global: 400–800 cycles" instead of using time.
When the parent writes "memory access latency can be 100–1000× longer than computation", it means: one arithmetic operation costs ~1 cycle, but one global-memory fetch costs hundreds of cycles. That ratio is the entire motivation for the hierarchy.
Definition Byte and its multiplesA byte is 8 bits , enough to store one small number or one character. Bigger stores are named by powers of 1024:
1 KB = 1024 1 MB = 1024 KB 1 GB = 1024 MB
A single float (a decimal number in the GPU) is 4 bytes — remember this number, the whole coalescing story uses it. Picture a byte as one small brick; a float is four bricks glued together.
A thread is one worker executing your program on one stream of data . Thousands run at once on a GPU. Each thread has a private notebook of scratch values — those live in registers (rung at the top of the ladder).
If this is new, build it up fully in GPU Thread Hierarchy . For this page you only need: a thread is one worker, and it owns some private storage.
A thread block is a team of threads that run on the same SM and can share a common scratchpad . That shared scratchpad is shared memory (the middle rung).
A warp is a group of exactly 32 threads that execute the same instruction together, in lockstep . This is the unit the memory system actually serves — you can't fetch for one thread, you fetch for a whole warp at once. See Warp Execution Model .
Why the topic needs the warp: coalescing and bank conflicts are both about "what happens when 32 threads ask for memory at the same instant". No warp, no coalescing.
Definition Streaming Multiprocessor (SM)An SM is one processing unit of the GPU — think of it as one factory floor. Each SM physically contains its own register file and its own shared memory. A GPU has many SMs. Because the fast rungs live inside an SM, they are small and must be split among all the threads running there.
This is the root of the whole register pressure story: the register file is a fixed-size cupboard bolted to the SM, and every resident thread must be given a drawer from it.
Now that the pictures exist, we can name the letters. Each entry is: symbol → plain words → the picture it lives in.
Definition Register-budget symbols
R t o t a l total number of registers on one SM (e.g. 65,536). Picture: the total number of drawers in the SM's cupboard.r registers used by one thread . Picture: drawers one worker occupies.N t h r e a d s how many threads can live on the SM at once , limited by drawers.N ma x the hardware ceiling on threads per SM (e.g. 2048), regardless of registers.⌊ ⌋ floor symbol: "round down to the nearest whole number", because you cannot run a fraction of a thread.
Occupancy is the fraction of the thread ceiling you actually fill :
Occupancy = N ma x N t h r e a d s N ma x Memory Latency Hiding ).
Definition Banking symbols
B number of banks in shared memory (typically 32). A bank is one narrow lane that serves one request per cycle.w bank width in bytes (typically 4 — exactly one float).a byte address , i.e. which byte you want.mod modulo operation: the remainder after division. 13 mod 5 = 3
Definition Segment and transaction symbols
Segment = an aligned block of memory (32/64/128 bytes) that global memory hands out as one unit. Picture floor tiles of fixed size; you always fetch whole tiles.N t r an s a c t i o n s how many distinct segments a warp's request touches . One tile → 1 transaction (fast). 32 scattered tiles → 32 transactions (32× slower). This is coalescing .⌈ ⌉ ceiling symbol: "round up ". Used because touching even 1 byte of a segment costs the whole segment.S worst crowd on any single bank , S = i max ( threads hitting bank i ) S
latency vs capacity ladder
shared memory and banks B w
global memory transactions
Read it top-down: the two units (time, size) build the ladder ; the worker hierarchy (thread → block → warp) plus the SM decide who shares which rung ; those together produce the three formula-families — registers/occupancy , bank conflicts , and coalescing — that make up the topic.
Worked example Reading the register formula onceAn SM has R t o t a l = 65 , 536 r = 40 N t h r e a d s = ⌊ 40 65 , 536 ⌋ = ⌊ 1638.4 ⌋ = 1638 Reading it: the cupboard holds 65,536 drawers, each thread wants 40, so 1638 complete threads fit and the leftover 16 drawers (65536 − 1638 × 40
Common mistake Floor is not "round to nearest"⌊ 1638.4 ⌋ = 1638 not 1638 rounded up. Storage that can't fill a whole thread is wasted, so we always round down for capacity-limited counts — and up (⌈ ⌉
Cover the right side and answer each aloud. If any fails, re-read its section before the next deep dive.
A clock cycle is one tick of the GPU's clock — the unit all latencies are counted in, so numbers stay comparable across clock speeds.
A float occupies how many bytes 4 bytes (and a bank is exactly 4 bytes wide, w = 4
A thread is one worker running your program on one data stream, with its own private registers.
A warp is exactly 32 threads executing in lockstep — the unit the memory system serves at once.
An SM is one processing floor of the GPU that physically holds the register file and shared memory, which is why they're small and shared.
R t o t a l r total registers on an SM, and registers used by one thread.
The ⌊ ⌋ round down to the nearest whole number, because you can't run a fraction of a thread.
Occupancy is N t h r e a d s / N ma x
A bank is one narrow lane of shared memory that serves one request per cycle; there are B = 32
The mod the remainder after division — it maps an address onto one of the 32 banks like hours wrap on a clock.
A segment is an aligned fixed-size block (32/64/128 B) that global memory hands out whole; N t r an s a c t i o n s
The ⌈ ⌉ round up — used for transactions because touching one byte costs the whole segment.
Next: with every symbol earned, you can now read the parent's three-level breakdown and its cousins CUDA Memory Types , Cache Architecture , and Matrix Multiplication Optimization without hitting an undefined letter.
