Intuition The ONE core ideaServing a large language model is slow for one reason: to write each new word, the machine must re-read billions of numbers and the memory of every earlier word — one word at a time . Everything in this topic (KV cache, PagedAttention, quantization) is a trick to move fewer bytes through the GPU's memory pipe so more users fit at once.
This page assumes nothing . If the parent note used a symbol, a word, or a picture without explaining it, we build it here first. Read this before the parent topic .
A token is a chunk of text — usually a word or a piece of a word — turned into a number the model can process. "unbelievable" might become 3 tokens: un, believ, able. The model never sees letters; it sees a sequence of token-numbers .
Think of a train. Each carriage is one token . The model reads the train left to right and, to add the next carriage, it must glance back at every carriage already on the track.
We will use these letters for counting tokens and model size — memorise the picture, not the letter :
Symbol
Plain words
The picture
S sequence length = how many tokens so far
number of train carriages
B batch = how many separate conversations at once
how many trains on parallel tracks
L number of layers in the model
how many times the train passes through the "thinking machine"
d m o d e l width of each token's vector
how many numbers describe one carriage
p bytes per number (FP16 = 2)
weight of storing one number
A vector is just a list of numbers . The model represents each token not as one number but as a long list — e.g. 5120 numbers for Llama-2-13B. That length is called d m o d e l
Look at the figure: one token (one carriage) is a tall stack of d m o d e l . When the parent note writes d m o d e l = 5120
Intuition Why length mattersBigger d m o d e l more bytes to store per token . That is exactly why d m o d e l
Definition Byte and precisionA byte is 8 bits — the basic unit of computer memory. A number's precision is how many bytes we spend storing it:
FP16 (half precision) = 2 bytes per number → p = 2 INT8 = 1 byte → p = 1 INT4 = half a byte → p = 0.5
Intuition The picture — a memory pipeImagine every number must travel through a fixed-width pipe (the memory bus) from storage to the compute chip. Fewer bytes per number = more numbers fit through the pipe per second. This single idea is the whole reason quantization exists — see Quantization Fundamentals .
The parent note keeps saying Keys , Values , queries . Here is where they come from, from zero. (Full detail lives in Attention Mechanism — we only build what this topic needs.)
Definition Query, Key, ValueWhen the model processes a token, it makes three lists from that token:
Query (q — "what am I looking for?"Key (k — "what do I offer to others searching?"Value (v — "the actual information I'll hand over if matched."
To decide how much attention token t j compares q t k j . Then it collects a weighted mix of the v j
Intuition Why the cache existsLook at the figure. To generate token 5, the query q 5 k 1 .. k 4 v 1 .. v 4 every earlier token . Those k v O ( t 2 ) store them. That stored pile of Keys and Values is the KV cache — and it is the memory hog the whole topic fights.
Definition Attention headThe model doesn't do attention once; it splits the token's d m o d e l heads , each attending independently, then re-joins them.
n h e a d d h e a d They fit together exactly: n h e a d ⋅ d h e a d = d m o d e l
Intuition Why the formula usesd m o d e l all heads gives back the full width d m o d e l d h e a d d m o d e l
A task is memory-bound moving data through the pipe , and compute-bound doing arithmetic . (More in GPU Memory & HBM Bandwidth and Throughput vs Latency Tradeoffs .)
Intuition The two phases, pictured
Prefill (reading the whole prompt): all prompt tokens go through the machine together in one big matrix multiply → the chip is busy, the pipe keeps up → compute-bound .Decode (writing one token): only one token's worth of work, but you still must drag all the weights and the whole KV cache through the pipe → the chip waits on the pipe → memory-bound .
This is the pivot of the entire topic: decode is memory-bound, so the game is move fewer bytes .
Quantization means replacing a fine-grained real number (needing 2 bytes) with the nearest value from a small set of allowed levels (needing half a byte), so it costs less to store and move. See Quantization Fundamentals , GPTQ and AWQ .
Intuition The picture — a ruler with few ticksA perfect ruler has infinitely many marks. INT4 gives you only 15 marks (2 4 − 1 scale s , and snapping causes an error of at most half a gap. That is why INT4 is smaller and slightly lossy — the levels are far apart.
s = 2 b − 1 r ma x − r min ( real units per tick, b = bits )
Here b number of bits , and 2 b − 1 number of gaps between 2 b b = 4 2 4 − 1 = 15
Batching = processing several conversations in the same pass so the weights (read once from the pipe) get reused across many users. More users per weight-read = higher throughput. See Batching Strategies .
Intuition Why it links to memoryEach conversation in the batch needs its own KV cache. So the batch size B how much KV-cache memory fits on the GPU . Waste less KV memory → bigger B
Token = numbered text chunk
Vector of d_model numbers
Query Key Value per token
KV cache stores past K and V
LLM Serving vLLM and Quantized Inference
Read the arrows as "you need this before that." Everything funnels into the parent topic node T .
A token is a number representing a chunk of text (word or word-piece).
A vector is an ordered list of numbers; each token is a vector of length d m o d e l
d m o d e l how many numbers describe one token.
S B L sequence length (tokens), batch (parallel conversations), number of layers.
p bytes per number — FP16 = 2, INT8 = 1, INT4 = 0.5.
Query / Key / Value are per-token lists: what I seek / what I offer / the info I hand over.
The KV cache exists to avoid recomputing past Keys and Values every decode step (trades memory for compute).
KV-cache size formula 2 ⋅ B ⋅ S ⋅ L ⋅ d m o d e l ⋅ p
Why d m o d e l d h e a d heads sum to d m o d e l
Memory-bound vs compute-bound slow part is moving data vs slow part is doing arithmetic.
Prefill is reading the whole prompt at once — compute-bound.
Decode is writing one token at a time — memory-bound.
Quantization is snapping real numbers to a small set of allowed levels to save bytes.
Scale s s = ( r ma x − r min ) / ( 2 b − 1 )
Why batching helps throughput weights read once are reused across many conversations.