LLM serving (vLLM, quantized inference)

WHY serving LLMs is hard

WHAT is the KV cache? In attention, the output for query $q_t$ needs Keys and Values of all previous tokens $1..t$ . Recomputing them every step is $O(t^2)$ wasted work, so we cache them. That cache is the memory hog.

The old way wasted memory (the problem vLLM solves)

Figure — LLM serving (vLLM, quantized inference)

HOW throughput jumps: less wasted memory → larger batch fits on the GPU → decode step processes more requests per weight-read → higher tokens/sec. vLLM reports up to ~24× throughput vs naive HF generate.

Quantized inference — shrinking the weights

Formula Affine (asymmetric) quantization — derived

We map real values $r$ in range $[r_{min}, r_{max}]$ to integers $q$ in $[0, 2^b-1]$ . Choose a linear map $r = s\,(q - z)$ :

scale $s = \dfrac{r_{max}-r_{min}}{2^b - 1}$ (real units per integer step)
zero-point $z = \text{round}\!\left(-\dfrac{r_{min}}{s}\right)$ (integer that maps to $0.0$ ) Then quantize / dequantize: $q = \text{clip}\!\left(\text{round}\!\left(\tfrac{r}{s}\right) + z,\ 0,\ 2^b-1\right), \qquad \hat r = s(q-z)$ Why round? integers are discrete — nearest integer minimizes error. Why clip? outliers beyond the range must saturate, not overflow. Symmetric variant: set $z=0$ , use range $[-r_{max}, r_{max}]$ ; simpler, common for weights.

Common mistakes

Flashcards

What are the two phases of LLM inference and which is memory-bound?

Prefill (process prompt, compute-bound) and Decode (one token at a time, memory-bound).

Why does the KV cache exist?

To avoid recomputing Keys/Values of all past tokens every decode step; trades memory for compute.

Formula for KV cache size in bytes.

2 \cdot B \cdot S \cdot L \cdot d_{model} \cdot p

(2 for K and V, p = bytes/element).

What OS concept does PagedAttention borrow?

Virtual memory / paging — non-contiguous fixed-size blocks mapped via a block table (page table analogy).

Two types of memory waste in naive KV allocation?

Internal fragmentation (reserved > used) and external fragmentation (free memory not contiguous).

What is continuous batching?

Injecting new requests and evicting finished ones every step so the GPU never idles on padding.

Affine quantization scale formula.

s = (r_{max}-r_{min})/(2^b-1)

Zero-point formula.

z = \text{round}(-r_{min}/s)

, the integer mapping to real 0.

What does W4A16 mean?

Weights in INT4, activations in FP16 (weight-only quantization).

Why does weight-only INT4 speed up decode but not necessarily prefill?

Decode is memory-bound (win from smaller HBM read); prefill is compute-bound and dequant-to-FP16 overhead can dominate.

Difference between GPTQ and AWQ?

GPTQ minimizes layer output error using Hessian info; AWQ protects salient weight channels chosen by activation magnitude.

Max per-request waste under PagedAttention?

At most block_size − 1 tokens (the partially-filled tail block).

Why is INT4 error much larger than INT8, not just 2×?

Step

s \propto 1/(2^b-1)

; 15 vs 255 levels → ~17× larger step.

Recall Feynman: explain to a 12-year-old

Imagine the AI writing a sentence one word at a time. To pick each next word it re-reads its whole memory of the conversation. That memory (the KV cache) is like a stack of sticky notes — one per word. Old programs grabbed a huge empty drawer for every user "just in case," so most drawers sat mostly empty and the desk filled up fast. vLLM instead uses small equal boxes and hands one out only when a note actually needs a home — no wasted space, so way more users share the desk. Quantization is like writing the AI's brain (its numbers) in shorthand: instead of long precise numbers, use short rounded ones. It's slightly less exact but you can flip through them much faster and carry more.

Connections

Attention Mechanism — KV cache comes directly from self-attention's K/V.
GPU Memory & HBM Bandwidth — why decode is memory-bound.
Quantization Fundamentals — affine map, scale/zero-point.
GPTQ and AWQ — PTQ algorithms for 4-bit weights.
Batching Strategies — static vs continuous/in-flight batching.
Model Sharding & Tensor Parallelism — complementary way to fit large models.
Throughput vs Latency Tradeoffs — what serving metrics you optimize.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, LLM ko serve karna basically memory ka game hai, compute ka nahi. Jab model output likhta hai to ek time pe ek hi token banata hai, aur har naye token ke liye usse purane saare tokens ko dobara "dekhna" padta hai. Isiliye hum KV cache rakhte hain — har purane token ke Key aur Value store karke, taaki baar-baar recompute na karna pade. Lekin yeh cache bahut bada ho jaata hai: ek 13B model, 4096 tokens, ek hi user ke liye ~3 GB kha jaata hai. Batch me 20 users? GPU OOM ho jaata hai.

Purane frameworks har request ke liye max length ka poora contiguous drawer reserve kar lete the, chahe user ne 10 hi token likhe hon. Isse 60-80% memory waste hoti thi (fragmentation). vLLM ne OS ka paging wala idea uthaya — KV cache ko chhote fixed blocks (16 tokens) me toda, aur ek block table se logical sequence ko non-contiguous physical blocks pe map kiya. Ab waste sirf last partial block tak seemit — matlab bigger batch fit hota hai, aur throughput 20x tak badh jaata hai. Saath me continuous batching hoti hai: finished request nikaal ke nayi ghusa dete hain, GPU kabhi khaali nahi baithti.

Doosri taraf quantization hai — weights ko FP16 (2 byte) se INT4/INT8 me chhota karna. Formula simple hai: $s = (r_{max}-r_{min})/(2^b-1)$ aur zero-point $z$ . Isse decode fast hota hai kyunki HBM se kam bytes read karne padte hain. Ek zaroori baat: INT4 weight-only me matmul se pehle wapas FP16 me dequantize karna padta hai, to speedup memory read se aata hai, arithmetic se nahi. Aur INT4 ki error INT8 se ~17x zyada hoti hai (15 vs 255 levels), isliye AWQ/GPTQ jaise smart methods salient weights ko protect karte hain. Bottom line: PagedAttention se memory bachao, quantization se weights chhota karo — dono milke ek hi GPU pe zyada users serve karte hain.

LLM serving (vLLM, quantized inference)

WHY serving LLMs is hard

The old way wasted memory (the problem vLLM solves)

Quantized inference — shrinking the weights

Common mistakes

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — MLOps & Deployment

Connections