LLM serving (vLLM, quantized inference): Visual walkthrough

Prerequisites we lean on (each is its own vault note): Attention Mechanism, GPU Memory & HBM Bandwidth, Batching Strategies, and the parent LLM serving.

Step 1 — One number is a few bytes

WHAT. The smallest thing a GPU stores is a single number — a weight, or one entry of a vector. A number in FP16 ("half precision") occupies exactly 2 bytes. We call this "bytes per element" and give it the name $p$ .

WHY start here. Every memory formula on this page is ultimately a count of numbers multiplied by bytes per number. If we never nail down "what is one number's cost", the big formula is just symbols. $p$ is the atom.

PICTURE. Look at the figure: the red box is one number. It is 2 tiny cells wide because FP16 = 2 bytes. Change the format and the box changes width — INT8 is 1 cell, INT4 is half a cell. That width is $p$ .

Figure — LLM serving (vLLM, quantized inference)

Step 2 — One token needs a Key vector and a Value vector

WHAT. In attention, to produce the next token the model asks a query and compares it against a Key for every earlier token, then mixes together their Values. So for each past token we must keep around two vectors: its Key ( $K$ ) and its Value ( $V$ ).

WHY two, not one. The Key answers "how relevant is this old token?" and the Value answers "what content does it contribute?". They play different roles, so both must be stored. That is the origin of the mysterious factor 2 in the parent's formula — it is literally "K and V".

PICTURE. One token stands as a red dot. From it hang two stacked bars: the K-bar and the V-bar. Each bar is a list of numbers (a vector). Store these once; never recompute.

Step 3 — Sum over heads: $d_{h e a d}$ becomes $d_{m o d e l}$

WHAT. Attention runs in parallel across several heads. If there are $n_{h e a d}$ heads and each carries a vector of length $d_{h e a d}$ , the total width per token is $n_{h e a d} \cdot d_{h e a d} = d_{m o d e l} .$ So instead of tracking heads separately, we collapse them into one wide number: $d_{m o d e l}$ , the model's "hidden width".

WHY this substitution. It makes the count clean. The parent wrote $d_{m o d e l}$ not $d_{h e a d}$ for exactly this reason — summing the head-vectors end-to-end reconstitutes the full hidden dimension.

PICTURE. Several short K-bars (one per head) laid side by side form one long bar of length $d_{m o d e l}$ . The red brace shows the total width.

Step 4 — Stack the layers: multiply by $L$

WHAT. A transformer is $L$ layers deep, and every layer runs its own attention, hence its own K and V. So one token's KV footprint repeats $L$ times.

WHY multiply, not add differently. Each layer is an independent copy of Step 3's cost; $L$ identical copies means multiply by $L$ .

PICTURE. The single-token bar from Step 3 is now photocopied into a tall stack of $L$ shelves. The red shelf is layer 40; the count is the same on every shelf.

Step 5 — Many tokens, many users: multiply by $S$ and $B$

WHAT. A conversation has $S$ tokens so far (sequence length), and the server handles $B$ conversations at once (batch size — see Batching Strategies). Every token of every request pays the Step-4 cost.

WHY these two multipliers. The KV cache stores every past token (that is the whole point — no recompute), so length $S$ multiplies in. And $B$ independent users means $B$ independent caches. Multiply both in, then attach $p$ from Step 1 to turn "count of numbers" into "bytes".

PICTURE. A grid: rows are the $S$ tokens of one conversation, and the grid is stamped $B$ times for $B$ users. The red column highlights one user's growing cache — it lengthens by one row per generated token.

Step 6 — The naive layout wastes most of it

WHAT. Old servers reserved a contiguous block sized to the maximum allowed length, e.g. 2048 tokens, for every request — even one that only reaches 100 tokens.

WHY it fails. The used part ( $S$ actual) is tiny; the reserved-but-empty tail is huge. That empty tail is internal fragmentation. Worse, when requests of different lengths come and go, the free memory left behind is scattered — external fragmentation — so a new request can't find one contiguous slab even when the total free memory is plenty.

PICTURE. A long horizontal bar reserved to max_len. Only the left red segment is used; the vast black-outlined empty region is wasted. Below, scattered free gaps illustrate external fragmentation.

Step 7 — PagedAttention: chop the cache into blocks

WHAT. Split the KV cache into small fixed-size blocks of, say, block_size = 16 tokens. Allocate a block only when the conversation actually reaches it. A block table (a little list) records where each block physically lives — the blocks need not be next to each other.

WHY this kills waste. You never reserve for tokens you don't have. The only leftover is the last, partly-filled block — at most block_size − 1 empty slots. Non-contiguous blocks also cure external fragmentation: any free block anywhere is usable.

PICTURE. The logical token sequence (red, contiguous) on top; arrows from the block table scatter it into physically separate blocks below. The single half-empty tail block is the only waste.

Step 8 — Edge & degenerate cases

WHAT / WHY / PICTURE, all in one figure with three panels:

$S$ exactly a multiple of block_size (e.g. $S = 32$ , block 16): the last block is full, waste $= 0$ . Best case — paging is free.
$S = 1$ (a single token): paging still grabs one whole block, so waste $= 15$ . This is paging's worst relative case, yet still tiny next to naive's ~2047.
Shared prefix (copy-on-write): two requests with the same system prompt point their block tables at the same physical blocks. Memory counted once, not twice — a saving the naive layout can never achieve.

The one-picture summary

This final figure compresses the whole journey: one number → one token's K+V → all heads → all layers → all tokens × all users builds the byte formula; then the same bytes are shown reserved-and-wasted (naive) versus paged-and-packed (vLLM), with the freed red memory refilled by extra batched users.

Recall Feynman retelling — say it back in plain words

Imagine remembering a conversation. For each thing anyone said, you keep two sticky notes: one saying how important it was (the Key) and one saying what it actually was (the Value). That's the "2".

Each sticky note is a little list of numbers as long as the model is wide — that width, once you glue all the parallel "heads" together, is $d_{m o d e l}$ . The model re-reads the conversation at every layer, so multiply by the number of layers $L$ . You keep a note for every word so far — that's $S$ — and you're doing this for every user at once — that's $B$ . Finally each number costs $p$ bytes. Multiply it all: $2 B S L d_{m o d e l} p$ . For one long Llama chat that's about 3 gigabytes.

The old servers were like renting a 2048-slot filing cabinet for every user even if they wrote 100 notes — 95% empty, and the empties scattered so nobody else could use them. PagedAttention instead hands out little 16-slot folders on demand, and writes down in a table where each folder sits. The only waste is the last half-full folder. Same 3 GB now holds far more conversations, so the GPU serves a bigger batch — and more batch means more tokens per second, because each expensive weight-read now feeds many users at once.

Recall Quick self-check

Where does the factor 2 in the KV formula come from? ::: One Key vector and one Value vector per token. Why $d_{m o d e l}$ instead of $d_{h e a d}$ ? ::: Summing over all $n_{h e a d}$ heads: $n_{h e a d} \cdot d_{h e a d} = d_{m o d e l}$ . Naive worst-case waste for max_len 2048, S 100? ::: $2048 - 100 = 1948$ tokens. Paged waste for that same request (block_size 16)? ::: $⌈ 100/16 ⌉ \cdot 16 - 100 = 12$ tokens. Does PagedAttention make the attention math faster? ::: No — it only changes memory layout so you can batch more; the FLOPs are unchanged.