LLM serving (vLLM, quantized inference): Exercises

Definition Prefill vs decode — the two phases every problem below depends on

Serving an LLM happens in two distinct phases, and almost every exercise here turns on which one you are in.

Prefill = the model reads your whole prompt at once and pushes all its tokens through the network in one big matmul. Because there is a mountain of arithmetic to do in parallel, prefill is compute-bound (the GPU's multipliers are the limit).
Decode = after the prompt, the model emits output tokens one at a time. Each step does only a sliver of arithmetic (a single new token) yet must re-read all the model weights from HBM plus the whole KV cache. Because the GPU sits waiting on memory, decode is memory-bound.

The picture below shows where those two phases spend the GPU's memory, and which lever attacks each. We refer back to it in L1.3, L4, and L5 — so read it now, don't just "keep it handy".

Figure — LLM serving (vLLM, quantized inference)

Definition Two more techniques this page uses — define them before we quiz them

Continuous (in-flight) batching = instead of forming a fixed batch and waiting for the slowest request to finish (which forces everyone else to sit on padding), the server evicts finished sequences and injects new ones every decode step. The GPU never idles waiting on padding, so utilisation stays high. It is orthogonal to PagedAttention but ships alongside it in vLLM. See Batching Strategies.
The W-x-A-y shorthand = a compact way to name what is quantized. W = weights, A = activations, and the number is the bit-width. So W4A16 = 4-bit weights, 16-bit (FP16) activations — the classic "weight-only" recipe that fixes memory-bound decode. W8A8 = 8-bit weights AND 8-bit activations, which additionally enables an INT8 matmul (a true compute speedup). We use these labels throughout Levels 2–5.

Level 1 — Recognition

Here we only recognise and recall the moving parts. No long derivations yet.

L1.1 — Classify each of the following as belonging to the prefill phase or the decode phase, and say whether it is compute-bound or memory-bound: (a) processing a 500-token prompt in one matmul; (b) generating the 501st token; (c) generating the 502nd token.

L1.2 — In the KV cache formula $KV bytes = 2 \cdot B \cdot S \cdot L \cdot d_{m o d e l} \cdot p$ , name in plain words what each of the six factors ( $2$ , $B$ , $S$ , $L$ , $d_{m o d e l}$ , $p$ ) stands for.

L1.3 — Using the intro figure's two buckets, match each technique to the bucket/bottleneck it attacks: techniques = {PagedAttention, INT4 weight-only quantization, continuous batching}; bottlenecks = {KV-cache memory waste (teal bucket), weight HBM read cost (orange bucket), GPU idling on padding}.

Recall Solution L1.1

(a) Prefill, compute-bound. All 500 prompt tokens are pushed through the network together in one big matmul. The GPU has tons of parallel arithmetic to chew on, so the limit is how fast it can multiply, not how fast it can fetch memory.

(b) and (c) Decode, memory-bound. After the prompt, tokens come out one at a time. Each step does only a sliver of arithmetic (a single new token) but must re-read all the model weights from HBM and read the whole KV cache. The GPU sits waiting on memory, not computing — hence memory-bound.

The dividing line is simply: many tokens at once ⇒ compute-bound (prefill); one token at a time ⇒ memory-bound (decode).

Recall Solution L1.2

$2$ — we store two vectors per token: a Key vector and a Value vector.
$B$ — batch size, how many separate requests share the GPU right now.
$S$ — sequence length, number of tokens whose K/V we have cached.
$L$ — number of layers; every transformer layer keeps its own K/V.
$d_{m o d e l}$ — hidden size; the length of each K (or V) vector once all heads are stacked, since $n_{h e a d} \cdot d_{h e a d} = d_{m o d e l}$ .
$p$ — bytes per element (2 for FP16, 1 for INT8), i.e. how heavy each stored number is.

Recall Solution L1.3

PagedAttention → KV-cache memory waste (teal bucket). It changes the layout of the KV cache into fixed-size blocks so almost nothing is reserved-but-unused.
INT4 weight-only quantization → weight HBM read cost (orange bucket). Smaller weights mean fewer bytes to fetch every decode step.
Continuous batching → GPU idling on padding. It swaps finished requests out and new ones in each step so the machine never waits on the slowest sequence.

Level 2 — Application

Now we plug numbers into the formulas we just recalled.

L2.1 — Compute the KV cache in gigabytes for a single request on a model with $L = 32$ layers, $d_{m o d e l} = 4096$ , sequence length $S = 2048$ , FP16 ( $p = 2$ ), batch $B = 1$ . (Use $1 GB = 1 0^{9}$ bytes.)

L2.2 — You have 40 GB of KV-cache budget on a GPU. Using the same model as L2.1 ( $L = 32$ , $d_{m o d e l} = 4096$ , $p = 2$ ) at $S = 2048$ per request, how many requests $B$ can you batch? (Take the floor.)

L2.3 — A 13-billion-parameter model. How much HBM do its weights take in FP16, in INT8, and in INT4? By what factor does the per-step weight read shrink going FP16 → INT4?

Recall Solution L2.1

Plug straight in: $2 \cdot 1 \cdot 2048 \cdot 32 \cdot 4096 \cdot 2 = 1.074 \times 1 0^{9} bytes \approx 1.07 GB .$ What we did: multiplied all six factors. Why: every stored K or V number costs $p$ bytes, and there are $2 \cdot B \cdot S \cdot L \cdot d_{m o d e l}$ of them. About 1 GB for one 2048-token conversation — this is why long chats are expensive.

Recall Solution L2.2

One request costs $1.074 \times 1 0^{9}$ bytes (from L2.1). The budget is $40 \times 1 0^{9}$ bytes. $B = ⌊ \frac{40 \times 1 0 ^{9}}{1.074 \times 1 0 ^{9}} ⌋ = ⌊ 37.3 ⌋ = 37.$ What we did: divided budget by per-request cost and floored (you cannot serve a fraction of a request). Why floor: the 38th request would overflow the budget and OOM.

Recall Solution L2.3

Weight bytes $= N \cdot p_{w}$ with $N = 13 \times 1 0^{9}$ :

FP16 ( $p_{w} = 2$ ): $26 \times 1 0^{9}$ bytes $= 26$ GB.
INT8 ( $p_{w} = 1$ ): $13 \times 1 0^{9}$ bytes $= 13$ GB.
INT4 ( $p_{w} = 0.5$ ): $6.5 \times 1 0^{9}$ bytes $= 6.5$ GB.

Shrink factor FP16 → INT4 $= 26/6.5 = 4 \times$ . Since decode is memory-bound, reading $4 \times$ fewer weight bytes per step is the whole point of weight-only quantization.

Level 3 — Analysis

Now we compare designs and reason about why one wins.

L3.1 (fragmentation) — A naive server reserves max_len = 2048 KV slots per request but a user only ends up using $120$ tokens. What fraction of that request's reserved KV memory is wasted? Now switch to PagedAttention with block_size = 16: what is the worst-case number of wasted tokens, and what wasted fraction does that give for the same 120-token request?

L3.2 (quantization error) — Weights lie in $[- 0.6, 0.9]$ . We use symmetric-style quantization here — meaning we describe error using the scale of the full range $s = (r_{ma x} - r_{min}) / (2^{b} - 1)$ , the "real units per integer step" (see definition callout below). Compare INT8 ( $b = 8$ ) vs INT4 ( $b = 4$ ) by comparing their scale $s$ . By what factor is the INT4 step larger than the INT8 step? Does that match the "INT4 is twice as bad" intuition?

L3.3 (where quantization does NOT help) — Explain, using the prefill/decode distinction, why W4A16 quantization can actually make large-batch prefill slower, even though it speeds up decode.

Recall Solution L3.1

Naive. Reserved $2048$ , used $120$ , wasted $2048 - 120 = 1928$ tokens. $waste fraction = \frac{1928}{2048} = 0.9414 \approx 94.1%.$ That is the internal-fragmentation disaster the parent note warns about.

Paged. Blocks hold 16 tokens each. $120$ tokens need $⌈ 120/16 ⌉ = 8$ blocks (since $7 \times 16 = 112 < 120 \leq 128 = 8 \times 16$ ). Those 8 blocks hold $128$ slots, so wasted $= 128 - 120 = 8$ tokens. The worst case for any request is block_size - 1 = 15 wasted tokens (only the tail block is ever under-full). $waste fraction here = \frac{8}{128} = 6.25%.$ What we did: counted only the tail-block slack. Why: paging allocates a block when needed, so every block except the last is full — waste is bounded by one block, independent of max_len.

Recall Solution L3.2

$r_{ma x} - r_{min} = 0.9 - (- 0.6) = 1.5$ .

INT8: $s_{8} = 1.5/ (2^{8} - 1) = 1.5/255 = 0.005882 \dots$
INT4: $s_{4} = 1.5/ (2^{4} - 1) = 1.5/15 = 0.1$

Ratio $s_{4} / s_{8} = 0.1/0.005882 = 17.0$ . The INT4 step is ~17× larger, and quantization error is bounded by $s /2$ , so INT4's error ceiling is ~17× INT8's — not 2×. The linear "twice as bad" intuition is wrong because levels scale as $2^{b} - 1$ : $255$ vs $15$ . This is exactly why 4-bit needs outlier tricks (GPTQ and AWQ, group-wise scales).

Recall Solution L3.3

Decode is memory-bound: it re-reads all weights per single-token step, so reading 4× fewer weight bytes (INT4) directly helps — quantization wins here.

Large-batch prefill is compute-bound: many prompt tokens flow through in one big matmul, and the GPU is busy multiplying, not waiting on weight memory. But W4A16 (4-bit weights, 16-bit activations) must first dequantize INT4 weights back to FP16 before that matmul. That dequant is extra work with no memory-read savings to pay for it (compute, not memory, was the limit). So the extra dequant step can make big-batch prefill slower. Fix if you want prefill speed too: use W8A8 (8-bit weights and 8-bit activations) so the matmul itself runs in INT8 (true compute speedup), rather than weight-only INT4.

Level 4 — Synthesis

Now combine KV cache, weights, and quantization into a single serving-fits decision. Keep the intro figure's two buckets (orange = weights, teal = KV) in mind — every part below just fills these buckets and checks against the 80 GB budget.

L4.1 (total footprint) — A GPU has 80 GB HBM. You serve a 13B model ( $N = 13 \times 1 0^{9}$ , $L = 40$ , $d_{m o d e l} = 5120$ ). You want $B = 16$ concurrent requests each at $S = 4096$ , with an FP16 KV cache ( $p = 2$ ) and FP16 weights ( $p_{w} = 2$ ). (a) Weight memory in FP16? (b) Total KV cache memory for all 16 requests? (c) Does it fit in 80 GB? If not, by how much do you overflow?

L4.2 (quantize to fit) — Take the failing case from L4.1. Quantize weights to INT4 ( $p_{w} = 0.5$ ), so this is a W4A16 setup — the KV cache stays FP16 ( $p = 2$ ). New weight memory? New total? Does it fit now, and with how much headroom?

L4.3 (rescue with more batch) — After INT4 weights (L4.2), how many extra requests $B$ could you add before running out of the 80 GB, given each request's FP16 KV cache ( $p = 2$ ) at $S = 4096$ costs the amount you computed per-request? (Floor the answer.)

Recall Solution L4.1

Parameters restated: $N = 13 \times 1 0^{9}$ , $L = 40$ , $d_{m o d e l} = 5120$ , $S = 4096$ , $B = 16$ , KV precision $p = 2$ (FP16), weight precision $p_{w} = 2$ (FP16).

(a) Weights FP16 (orange bucket): $N \cdot p_{w} = 13 \times 1 0^{9} \cdot 2 = 26 \times 1 0^{9}$ bytes $= 26$ GB.

(b) KV cache (teal bucket), $B = 16$ : apply $2 \cdot B \cdot S \cdot L \cdot d_{m o d e l} \cdot p$ . First one request ( $B = 1$ ): $2 \cdot 1 \cdot 4096 \cdot 40 \cdot 5120 \cdot 2 = 3.355 \times 1 0^{9}$ bytes $\approx 3.36$ GB. Times 16: $16 \cdot 3.355 \times 1 0^{9} = 5.369 \times 1 0^{10} bytes \approx 53.69 GB .$

(c) Total: $26 + 53.69 = 79.69$ GB. Purely by the two formulas this barely squeaks under ( $79.69 < 80$ ), leaving only $0.31$ GB — but this ignores activations, CUDA context, and fragmentation overhead, which cost several GB in practice. So it is effectively an OOM: treat it as "does not safely fit."

Recall Solution L4.2

Same model, but now INT4 weights ( $p_{w} = 0.5$ ), KV cache unchanged at FP16 ( $p = 2$ ). This is a W4A16 configuration (4-bit weights, 16-bit activations).

New weight memory (orange bucket shrinks): $N \cdot p_{w} = 13 \times 1 0^{9} \cdot 0.5 = 6.5 \times 1 0^{9} bytes = 6.5 GB$ (down from 26 GB — saved 19.5 GB).

KV cache unchanged (teal bucket): still $53.69$ GB, because weight-only quantization does not touch the KV cache.

New total: $6.5 + 53.69 = 60.19$ GB.

Fit? Yes: $60.19 < 80$ , headroom $= 80 - 60.19 = 19.81$ GB. Quantizing weights bought back almost 20 GB — plenty of room now for activations and more batch.

Recall Solution L4.3

Free space after L4.2 $= 19.81$ GB $= 19.81 \times 1 0^{9}$ bytes. Each request's FP16 KV cache ( $p = 2$ ) $= 3.355 \times 1 0^{9}$ bytes (from L4.1(b), the $B = 1$ value). $Δ B = ⌊ \frac{19.81 \times 1 0 ^{9}}{3.355 \times 1 0 ^{9}} ⌋ = ⌊ 5.90 ⌋ = 5.$ So you could add 5 more requests ( $B$ from 16 to 21) before the KV cache eats the headroom. This is the whole story of the parent note: shrink weights → free HBM → bigger batch → more tokens/sec.

Level 5 — Mastery

One open-ended decision problem that forces you to pick the right tool for the right bottleneck. Refer back to the intro figure: your first move is always to ask which bucket (or the padding-idle problem) is the wall here?

L5.1 (diagnose and choose) — Two production scenarios. For each, identify the dominant bottleneck and choose the single most effective technique from {PagedAttention, continuous batching, W4A16 weight-only quantization, W8A8 quantization}. Justify with the formulas.

Scenario A: A chatbot with thousands of short sessions (average 60 tokens each), but the server reserves max_len = 2048 KV slots per request. The GPU keeps OOMing even though average usage is tiny. Which bucket is overflowing, and which single technique fixes it?
Scenario B: A batch document-summarisation job: huge prompts (many thousands of tokens each) processed at large batch size. Latency does not matter, but you want maximum tokens/sec throughput, and the model barely fits on the GPU. Which phase dominates, which bottleneck follows, and which single technique helps most?

Recall Solution L5.1

Scenario A → PagedAttention. Diagnose: the overflowing bucket is the teal KV cache, and the cause is memory waste, not weight cost. Each request reserves 2048 slots but uses 60, wasting $\frac{2048 - 60}{2048} = 97%$ of its KV allocation (internal fragmentation). The GPU OOMs on reserved-but-empty KV memory. Fix: PagedAttention allocates 16-token blocks on demand, so a 60-token session uses $⌈ 60/16 ⌉ = 4$ blocks $= 64$ slots — waste drops to $4$ tokens ( $6.25%$ ). Freed memory ⇒ far larger batch ⇒ no OOM. (Continuous batching is a great complement but the root cause here is fragmentation.)

Scenario B → W8A8 quantization. Diagnose: huge prompts at large batch means prefill dominates, and prefill is compute-bound. Weight-only INT4 (W4A16) would not help — it even adds dequant overhead in compute-bound prefill (see L3.3). You also want the model to fit more comfortably. Fix: W8A8 (8-bit weights and 8-bit activations) halves weight bytes (helps it fit) and runs the matmul in INT8 for a genuine compute speedup — exactly what a compute-bound prefill workload needs. Since latency doesn't matter, you can also crank batch size for throughput. Do not reach for PagedAttention as the primary lever here: prompts are long, so there is little tail-block waste to reclaim relative to the compute cost.

Master takeaway: first ask "is this memory-bound (decode/KV) or compute-bound (large-batch prefill)?", then pick the lever that attacks that wall — memory tools (PagedAttention, W4A16) for memory walls, INT8 matmul (W8A8) for compute walls.

Quick self-check

Bottleneck of the decode phase

Memory-bound — reads all weights + full KV cache per single token.

Worst-case wasted tokens per request under PagedAttention with block_size B

B - 1

(only the tail block is under-full).

Ratio of INT4 to INT8 quantization step for the same value range

255/15 \approx 17 \times

larger step at INT4.

What does W4A16 mean

4-bit weights, 16-bit (FP16) activations — the weight-only recipe that fixes memory-bound decode.

Which lever attacks compute-bound large-batch prefill

W8A8 (INT8 matmul gives a true compute speedup); weight-only INT4 does not.

Does INT4 weight-only quantization shrink the KV cache

No — it only shrinks weights (orange bucket); KV cache (teal) stays FP16 unless separately quantized.