LLM serving (vLLM, quantized inference): Question bank

Definition Every symbol used on this page, in plain words

Before any trap, here is the whole vocabulary — nothing below uses a symbol not listed here.

Prefill = the phase where the model reads your entire prompt at once in one big parallel matmul (matrix multiplication). Lots of arithmetic, done in one shot.
Decode = the phase where the model produces the answer one token at a time (a token ≈ a word-piece). Each step does a tiny bit of work but must re-read everything.
HBM bandwidth = how many bytes per second the GPU can pull weights out of its fast on-chip memory (HBM = High-Bandwidth Memory). See GPU Memory & HBM Bandwidth.
KV cache = the stored Keys ( $K$ ) and Values ( $V$ ) of every past token, so attention doesn't recompute them each step.
Continuous batching = the scheduler's habit of removing finished requests and slotting in new ones every single decode step, so the GPU never idles waiting for the slowest one.
In the KV-cache formula $2 B S L d_{m o d e l} p$ : $B$ = number of requests batched together, $S$ = sequence length (tokens so far), $L$ = number of transformer layers, $d_{m o d e l}$ = width of each token's vector, $p$ = bytes per number (2 for FP16). The leading $2$ = one copy for $K$ , one for $V$ .
$t$ = a running token index; $O (t^{2})$ means "cost grows with the square of how many tokens you've seen."
$n_{h e a d}$ = number of attention heads; $d_{h e a d}$ = width of one head. They multiply to $d_{m o d e l}$ .
block_size = how many tokens' worth of KV live in one fixed-size PagedAttention block (e.g. 16).
In quantization: $r$ = a real weight value, $r_{min}, r_{ma x}$ = the smallest/largest real value in the range, $b$ = number of bits, $q$ = the stored integer, $z$ = zero-point (the integer that decodes back to real $0.0$ ), $s$ = step size / scale = real-value gap between two neighbouring integers, $\overset{r}{^}$ = the value you recover after rounding.

Before you start, hold three anchors in your head:

Decode is memory-bound — the clock is set by how fast weights + KV cache stream out of HBM, not by arithmetic.
PagedAttention changes where memory lives, never what math attention does.
Quantization error is set by the step size $s$ — and the next callout derives exactly why $s$ shrinks like $1/ (2^{b} - 1)$ .

True or false — justify

PagedAttention makes the attention computation itself run fewer FLOPs.

False. The dot-products and softmax are identical; PagedAttention only changes the KV memory layout so larger batches fit — throughput rises from batching, not faster math.

Weight-only INT4 (W4A16) always speeds up the prefill phase.

False. Prefill is compute-bound with big matmuls; the mandatory INT4→FP16 dequant adds overhead that can make prefill slower. The win is in memory-bound decode.

The KV cache grows quadratically with sequence length.

False. The stored cache is linear in

S

(

2 B S L d_{m o d e l} p

); the quadratic

O (t^{2})

cost is the recomputation you avoid by caching.

Continuous batching and PagedAttention are the same mechanism.

False. PagedAttention manages KV allocation; continuous batching manages scheduling (evict finished, inject new each step). They ship together but are orthogonal.

Symmetric quantization (

z = 0

) is strictly worse than asymmetric because it has a fixed zero-point.

False. For weights that are roughly zero-centred, symmetric wastes no range and is simpler/faster; asymmetric only helps when the value range is lopsided (e.g. ReLU activations).

Making block_size as large as possible minimizes bookkeeping and is therefore best.

False. Large blocks re-introduce internal fragmentation — the tail block wastes up to block_size − 1 tokens per request. The sweet spot is ~16–32.

INT4 has exactly twice the accuracy loss of INT8.

False. Error scales with step

s \propto 1/ (2^{b} - 1)

: INT8 has 255 gaps, INT4 has 15, so the step is ~17× larger — much worse than 2×.

Quantizing weights lets you skip storing an FP16 copy at inference time.

True (for the stored weights) — you keep only the INT packed weights plus tiny per-group scales; the FP16 values are reconstructed on the fly per matmul, never persisted.

If a GPU has spare HBM, bigger batches always increase tokens/sec.

False. Only until you hit the bandwidth ceiling — once decode saturates HBM, adding requests raises latency without raising throughput (see Throughput vs Latency Tradeoffs).

Two requests sharing the same system prompt must each store their own copy of that prompt's KV.

False. PagedAttention lets them share the identical prefix blocks via copy-on-write; a private copy is made only when their generations diverge.

Spot the error

"KV cache size =

B \cdot S \cdot L \cdot d_{m o d e l} \cdot p

Missing the factor 2 — you store both Keys and Values per token, so it's

2 B S L d_{m o d e l} p

"We use

d_{h e a d}

in the KV formula because attention works per head."

Wrong dimension — you sum over all heads, and

n_{h e a d} \cdot d_{h e a d} = d_{m o d e l}

, so the whole-cache formula uses

d_{m o d e l}

"AWQ works by minimizing the layer's output error with the Hessian."

That's GPTQ. AWQ protects salient weight channels identified by activation magnitude — no Hessian.

"W8A8 only quantizes weights, so it's identical to W4A16 but at 8 bits."

W8A8 quantizes activations too, enabling native INT8 matmul (compute speedup); W4A16 keeps activations FP16 and only cuts the HBM read.

"Zero-point

z = round (r_{min} / s)

Sign error — it's

z = round (- r_{min} / s)

, the integer that maps back to real

0.0

under

\overset{r}{^} = s (q - z)

"Per-tensor quantization is finer-grained than per-channel."

Backwards. Per-tensor uses one

(s, z)

for everything (coarsest); per-channel/per-group uses many scales (finer, lower error).

"PagedAttention eliminates all KV memory waste."

Not all — the tail block is still partly empty, so waste is bounded by block_size − 1 tokens per request, just far smaller than naive contiguous reservation.

"The naive framework fragments memory because attention is

O (t^{2})

Unrelated causes. Fragmentation comes from reserving max_seq_len contiguous blocks per request, not from the attention cost.

Why questions

Why is decode memory-bound while prefill is compute-bound?

Prefill runs all prompt tokens through one big parallel matmul (lots of arithmetic per weight-read); decode does one token of work while still reading all weights + KV, so bandwidth dominates.

Why does reducing weights to INT4 speed up decode even though the matmul still runs in FP16?

The bottleneck is reading weights from HBM; INT4 is 4× fewer bytes to fetch, and dequant is cheap relative to that saved transfer — arithmetic isn't the limiter.

Why does quantization use rounding and clipping rather than just one?

Rounding maps a real value to the nearest representable integer (bounds error by

s /2

); clipping forces out-of-range outliers to saturate at the extremes instead of wrapping/overflowing.

Why do 4-bit methods rely so heavily on group-wise scales and outlier handling?

With only 15 gaps the step

s

is large, so a single per-tensor scale bloated by one outlier ruins every other weight; per-group scales keep the step tight where the values actually cluster.

Why can PagedAttention raise throughput ~24× yet not touch latency of a single request?

It packs more requests into the same HBM (bigger batch → more useful work per weight-read), but one request's own token-by-token generation path is unchanged.

Why does tensor parallelism not remove the memory-bound nature of decode?

Sharding splits weights and KV across GPUs to fit larger models, but each GPU still streams its shard from HBM every step plus adds cross-GPU communication — the per-step bottleneck stays bandwidth.

Edge cases

What is the KV waste for a request whose length is an exact multiple of block_size?

Zero tail waste — every block is completely full, the best case for PagedAttention.

What happens to the quantization error for a weight that lands exactly halfway between two levels?

It hits the maximum per-element error of $s /2$ ; rounding must pick one side, and half a step is the worst case.

What is the quantization output for a real value far above

r_{ma x}

It clips to the top integer

2^{b} - 1

; dequantizing gives

\overset{r}{^} = r_{ma x}

(the outlier saturates rather than overflowing).

What is the quantization output for a real value far below

r_{min}

It clips to the bottom integer

0

; dequantizing gives

\overset{r}{^} = s (0 - z) = r_{min}

, the mirror-image saturation at the lower bound.

With batch size 1 and a short prompt, does PagedAttention help throughput?

Barely — its win comes from packing many requests; with one small request there's little fragmentation to reclaim, so gains are minimal.

What happens if two requests sharing a prefix each try to write a new token into a shared block?

Copy-on-write triggers — the shared block is duplicated so each request writes to its own copy, preserving isolation.

If block_size = 1, what have you effectively built?

Per-token allocation with zero tail waste but maximal block-table overhead and poor kernel efficiency — the opposite extreme from a huge block.

What is the KV cache size when generation is at the very first decode step (

S = 1

Just one token's worth:

2 B L d_{m o d e l} p

— the cache starts tiny and grows linearly as tokens accumulate.

Recall One-sentence self-test

Cover the answers and explain: why does quantization win on decode but risk losing on prefill? Answer ::: Decode is memory-bound so fewer weight bytes = faster; prefill is compute-bound so the extra dequant work can outweigh the smaller read.