LLM serving (vLLM, quantized inference): Exercises

5.3.18 · D4 · AI-ML › MLOps & Deployment › LLM serving (vLLM, quantized inference)

Definition Prefill vs decode — woh do phases jinpar neeche har problem depend karti hai

Ek LLM serve karna do alag phases mein hota hai, aur yahan ka almost har exercise inhi mein se kisi ek par tikta hai.

Prefill = model tumhara poora prompt ek baar mein padhta hai aur saare tokens ko network ke through ek bade matmul mein push karta hai. Kyunki parallel mein bahut saara arithmetic karna hota hai, prefill compute-bound hoti hai (GPU ke multipliers limit hote hain).
Decode = prompt ke baad, model output tokens ek ek karke emit karta hai. Har step mein sirf thoda sa arithmetic hota hai (ek naya token), lekin ushe HBM se saare model weights aur poora KV cache re-read karna padta hai. Kyunki GPU memory ka wait kar raha hota hai, decode memory-bound hoti hai.

Neeche wali picture dikhati hai ki woh do phases GPU ki memory kahan kharach karte hain, aur kaun sa lever kise attack karta hai. Hum isko L1.3, L4, aur L5 mein refer karte hain — toh ise abhi padho, sirf "haath ke paas rakhna" mat socho.

Figure — LLM serving (vLLM, quantized inference)

Definition Do aur techniques jinhe yeh page use karta hai — quiz se pehle inhe define karte hain

Continuous (in-flight) batching = ek fixed batch banana aur sabse slow request ke khatam hone ka wait karne ki jagah (jo baaki sabko padding par bithne par majboor karta hai), server har decode step par finished sequences ko evict karke nayi ones inject karta hai. GPU kabhi padding ka wait nahi karta, toh utilisation high rehti hai. Yeh PagedAttention se orthogonal hai lekin vLLM mein uske saath ship hota hai. Dekho Batching Strategies.
The W-x-A-y shorthand = kya quantize hua hai yeh batane ka ek compact tarika. W = weights, A = activations, aur number bit-width hai. Toh W4A16 = 4-bit weights, 16-bit (FP16) activations — classic "weight-only" recipe jo memory-bound decode ko fix karta hai. W8A8 = 8-bit weights AUR 8-bit activations, jo additionally ek INT8 matmul enable karta hai (ek sach mein compute speedup). Hum yeh labels Level 2–5 mein use karte hain.

Level 1 — Recognition

Yahan hum sirf moving parts ko pehchante aur recall karte hain. Abhi koi lambi derivations nahi.

L1.1 — Neeche diye gaye har item ko prefill phase ya decode phase mein classify karo, aur batao ki yeh compute-bound hai ya memory-bound: (a) 500-token prompt ko ek matmul mein process karna; (b) 501va token generate karna; (c) 502va token generate karna.

L1.2 — KV cache formula $KV bytes = 2 \cdot B \cdot S \cdot L \cdot d_{m o d e l} \cdot p$ mein, chhe factors ( $2$ , $B$ , $S$ , $L$ , $d_{m o d e l}$ , $p$ ) mein se har ek ka plain words mein matlab batao.

L1.3 — Intro figure ke do buckets ka use karke, har technique ko us bucket/bottleneck se match karo jise woh attack karta hai: techniques = {PagedAttention, INT4 weight-only quantization, continuous batching}; bottlenecks = {KV-cache memory waste (teal bucket), weight HBM read cost (orange bucket), GPU idling on padding}.

Recall Solution L1.1

(a) Prefill, compute-bound. Saare 500 prompt tokens ek saath network se together ek bade matmul mein guzarte hain. GPU ke paas tons of parallel arithmetic hoti hai, toh limit yeh hai ki woh kitni jaldi multiply kar sakta hai, na ki memory fetch karne mein.

(b) aur (c) Decode, memory-bound. Prompt ke baad, tokens ek ek karke nikalte hain. Har step mein sirf thoda sa arithmetic hota hai (ek naya token) lekin ushe HBM se saare model weights aur poora KV cache re-read karna padta hai. GPU computing ki jagah memory ka wait karta rehta hai — isliye memory-bound.

Dividing line simple hai: ek saath kaafi tokens ⇒ compute-bound (prefill); ek ek token ⇒ memory-bound (decode).

Recall Solution L1.2

$2$ — hum har token ke liye do vectors store karte hain: ek Key vector aur ek Value vector.
$B$ — batch size, abhi GPU par kitne alag requests share kar rahe hain.
$S$ — sequence length, kitne tokens ke K/V hum ne cache kiye hain.
$L$ — layers ki sankhya; har transformer layer apna K/V rakhti hai.
$d_{m o d e l}$ — hidden size; har K (ya V) vector ki length jab saare heads stack ho jaate hain, kyunki $n_{h e a d} \cdot d_{h e a d} = d_{m o d e l}$ .
$p$ — bytes per element (FP16 ke liye 2, INT8 ke liye 1), yaani har stored number kitna heavy hai.

Recall Solution L1.3

PagedAttention → KV-cache memory waste (teal bucket). Yeh KV cache ka layout fixed-size blocks mein change karta hai taaki almost kuch bhi reserved-but-unused na rahe.
INT4 weight-only quantization → weight HBM read cost (orange bucket). Chhote weights matlab har decode step mein fetch karne ke liye kam bytes.
Continuous batching → GPU idling on padding. Yeh har step mein finished requests bahar karta hai aur nayi andar laata hai taaki machine kabhi slowest sequence ka wait na kare.

Level 2 — Application

Ab hum un formulas mein numbers plug karte hain jo humne abhi recall kiye.

L2.1 — $L = 32$ layers, $d_{m o d e l} = 4096$ , sequence length $S = 2048$ , FP16 ( $p = 2$ ), batch $B = 1$ wale ek model par ek single request ke liye KV cache gigabytes mein compute karo. ( $1 GB = 1 0^{9}$ bytes use karo.)

L2.2 — Ek GPU par tumhare paas 40 GB KV-cache budget hai. L2.1 wale same model ( $L = 32$ , $d_{m o d e l} = 4096$ , $p = 2$ ) ko $S = 2048$ per request par use karte hue, tum kitne requests $B$ batch kar sakte ho? (Floor lo.)

L2.3 — 13 billion parameters wala model. Iske weights FP16, INT8, aur INT4 mein kitna HBM lete hain? FP16 → INT4 jaate waqt per-step weight read kis factor se shrink hoti hai?

Recall Solution L2.1

Seedha plug in karo: $2 \cdot 1 \cdot 2048 \cdot 32 \cdot 4096 \cdot 2 = 1.074 \times 1 0^{9} bytes \approx 1.07 GB .$ Humne kya kiya: saare chhe factors multiply kiye. Kyun: har stored K ya V number $p$ bytes cost karta hai, aur unki sankhya $2 \cdot B \cdot S \cdot L \cdot d_{m o d e l}$ hai. Ek 2048-token conversation ke liye lagbhag 1 GB — yahi wajah hai ki lambi chats mehngi hoti hain.

Recall Solution L2.2

Ek request ka cost $1.074 \times 1 0^{9}$ bytes (L2.1 se). Budget hai $40 \times 1 0^{9}$ bytes. $B = ⌊ \frac{40 \times 1 0 ^{9}}{1.074 \times 1 0 ^{9}} ⌋ = ⌊ 37.3 ⌋ = 37.$ Humne kya kiya: budget ko per-request cost se divide kiya aur floor liya (tum ek fraction request serve nahi kar sakte). Floor kyun: 38va request budget overflow karke OOM dega.

Recall Solution L2.3

Weight bytes $= N \cdot p_{w}$ jahan $N = 13 \times 1 0^{9}$ :

FP16 ( $p_{w} = 2$ ): $26 \times 1 0^{9}$ bytes $= 26$ GB.
INT8 ( $p_{w} = 1$ ): $13 \times 1 0^{9}$ bytes $= 13$ GB.
INT4 ( $p_{w} = 0.5$ ): $6.5 \times 1 0^{9}$ bytes $= 6.5$ GB.

Shrink factor FP16 → INT4 $= 26/6.5 = 4 \times$ . Kyunki decode memory-bound hai, $4 \times$ kam weight bytes per step padhna weight-only quantization ka poora point hai.

Level 3 — Analysis

Ab hum designs compare karte hain aur sochte hain ki ek kyun jeetta hai.

L3.1 (fragmentation) — Ek naive server max_len = 2048 KV slots per request reserve karta hai lekin user sirf $120$ tokens use karta hai. Us request ki reserved KV memory ka kitna fraction waste hota hai? Ab PagedAttention par switch karo jahan block_size = 16 hai: worst-case mein kitne tokens waste hote hain, aur usi 120-token request ke liye woh kaunsa wasted fraction deta hai?

L3.2 (quantization error) — Weights $[- 0.6, 0.9]$ mein hain. Hum yahan symmetric-style quantization use karte hain — matlab hum error ko full range ke scale $s = (r_{ma x} - r_{min}) / (2^{b} - 1)$ se describe karte hain, yaani "real units per integer step" (neeche definition callout dekho). INT8 ( $b = 8$ ) vs INT4 ( $b = 4$ ) ko unke scale $s$ compare karke compare karo. INT4 step INT8 step se kitne factor bada hai? Kya yeh "INT4 twice as bad hai" wali intuition se match karta hai?

L3.3 (jahan quantization help NAHI karta) — Prefill/decode distinction use karke explain karo ki kyun W4A16 quantization actually large-batch prefill ko slower bana sakta hai, bhaale hi yeh decode ko speed up karta ho.

Recall Solution L3.1

Naive. Reserve kiya $2048$ , use kiya $120$ , waste $2048 - 120 = 1928$ tokens. $waste fraction = \frac{1928}{2048} = 0.9414 \approx 94.1%.$ Yahi woh internal-fragmentation disaster hai jiske baare mein parent note warn karta hai.

Paged. Blocks 16 tokens rakhte hain. $120$ tokens ke liye $⌈ 120/16 ⌉ = 8$ blocks chahiye (kyunki $7 \times 16 = 112 < 120 \leq 128 = 8 \times 16$ ). Woh 8 blocks $128$ slots rakhte hain, toh waste $= 128 - 120 = 8$ tokens. Kisi bhi request ke liye worst case hai block_size - 1 = 15 wasted tokens (sirf tail block hi ever under-full hota hai). $waste fraction yahan = \frac{8}{128} = 6.25%.$ Humne kya kiya: sirf tail-block slack count kiya. Kyun: paging ek block zarurat padne par allocate karta hai, toh last ke alawa har block full hota hai — waste ek block se bounded hai, max_len se independent.

Recall Solution L3.2

$r_{ma x} - r_{min} = 0.9 - (- 0.6) = 1.5$ .

INT8: $s_{8} = 1.5/ (2^{8} - 1) = 1.5/255 = 0.005882 \dots$
INT4: $s_{4} = 1.5/ (2^{4} - 1) = 1.5/15 = 0.1$

Ratio $s_{4} / s_{8} = 0.1/0.005882 = 17.0$ . INT4 step ~17× bada hai, aur quantization error $s /2$ se bounded hai, toh INT4 ka error ceiling INT8 ka ~17× hai — 2× nahi. Linear "twice as bad" intuition galat hai kyunki levels $2^{b} - 1$ ke saath scale karte hain: $255$ vs $15$ . Yahi exact wajah hai ki 4-bit ko outlier tricks ki zarurat hai (GPTQ and AWQ, group-wise scales).

Recall Solution L3.3

Decode memory-bound hai: yeh single-token step per saare weights re-read karta hai, toh 4× kam weight bytes (INT4) padhna directly help karta hai — quantization yahan jeetta hai.

Large-batch prefill compute-bound hai: kaafi prompt tokens ek bade matmul mein ek saath flow karte hain, aur GPU multiply karne mein busy hota hai, weight memory ka wait nahi karta. Lekin W4A16 (4-bit weights, 16-bit activations) ko us matmul se pehle INT4 weights ko FP16 mein dequantize karna padta hai. Woh dequant extra kaam hai jiske liye koi memory-read savings nahi hain (compute, na ki memory, limit tha). Toh extra dequant step big-batch prefill ko slower bana sakta hai. Agar prefill speed bhi chahiye: W8A8 (8-bit weights aur 8-bit activations) use karo taaki matmul khud INT8 mein chale (sach mein compute speedup), weight-only INT4 ki jagah.

Level 4 — Synthesis

Ab KV cache, weights, aur quantization ko ek single serving-fits decision mein combine karo. Intro figure ke do buckets (orange = weights, teal = KV) dhyan mein rakho — neeche ka har part bas inhi buckets ko bharta hai aur 80 GB budget se compare karta hai.

L4.1 (total footprint) — Ek GPU mein 80 GB HBM hai. Tum ek 13B model ( $N = 13 \times 1 0^{9}$ , $L = 40$ , $d_{m o d e l} = 5120$ ) serve kar rahe ho. Tum $B = 16$ concurrent requests chahte ho har ek $S = 4096$ par, FP16 KV cache ( $p = 2$ ) aur FP16 weights ( $p_{w} = 2$ ) ke saath. (a) FP16 mein weight memory? (b) Saare 16 requests ke liye total KV cache memory? (c) Kya yeh 80 GB mein fit hoga? Agar nahi, toh kitna overflow hoga?

L4.2 (fit karne ke liye quantize karo) — L4.1 wala failing case lo. Weights ko INT4 ( $p_{w} = 0.5$ ) par quantize karo, toh yeh W4A16 setup hai — KV cache FP16 ( $p = 2$ ) rehta hai. Naya weight memory? Naya total? Kya yeh ab fit hota hai, aur kitna headroom hai?

L4.3 (zyada batch se rescue) — INT4 weights ke baad (L4.2), 80 GB se bahar jaane se pehle tum kitne extra requests $B$ add kar sakte ho, given ki har request ki FP16 KV cache ( $p = 2$ ) $S = 4096$ par itni cost karti hai jitni tumne per-request compute ki? (Answer floor karo.)

Recall Solution L4.1

Parameters: $N = 13 \times 1 0^{9}$ , $L = 40$ , $d_{m o d e l} = 5120$ , $S = 4096$ , $B = 16$ , KV precision $p = 2$ (FP16), weight precision $p_{w} = 2$ (FP16).

(a) Weights FP16 (orange bucket): $N \cdot p_{w} = 13 \times 1 0^{9} \cdot 2 = 26 \times 1 0^{9}$ bytes $= 26$ GB.

(b) KV cache (teal bucket), $B = 16$ : $2 \cdot B \cdot S \cdot L \cdot d_{m o d e l} \cdot p$ apply karo. Pehle ek request ( $B = 1$ ): $2 \cdot 1 \cdot 4096 \cdot 40 \cdot 5120 \cdot 2 = 3.355 \times 1 0^{9}$ bytes $\approx 3.36$ GB. 16 se multiply karo: $16 \cdot 3.355 \times 1 0^{9} = 5.369 \times 1 0^{10} bytes \approx 53.69 GB .$

(c) Total: $26 + 53.69 = 79.69$ GB. Sirf do formulas se yeh muskil se andar aata hai ( $79.69 < 80$ ), sirf $0.31$ GB bachta hai — lekin yeh activations, CUDA context, aur fragmentation overhead ignore karta hai, jo practice mein kaafi GBs cost karte hain. Toh yeh effectively ek OOM hai: ise "safely fit nahi hota" treat karo.

Recall Solution L4.2

Same model, lekin ab INT4 weights ( $p_{w} = 0.5$ ), KV cache FP16 ( $p = 2$ ) par unchanged. Yeh ek W4A16 configuration hai (4-bit weights, 16-bit activations).

Naya weight memory (orange bucket shrinks): $N \cdot p_{w} = 13 \times 1 0^{9} \cdot 0.5 = 6.5 \times 1 0^{9} bytes = 6.5 GB$ (26 GB se kam — 19.5 GB bacha).

KV cache unchanged (teal bucket): abhi bhi $53.69$ GB, kyunki weight-only quantization KV cache ko touch nahi karta.

Naya total: $6.5 + 53.69 = 60.19$ GB.

Fit? Haan: $60.19 < 80$ , headroom $= 80 - 60.19 = 19.81$ GB. Weights quantize karne se almost 20 GB wapas mila — activations aur zyada batch ke liye ab kaafi room hai.

Recall Solution L4.3

L4.2 ke baad free space $= 19.81$ GB $= 19.81 \times 1 0^{9}$ bytes. Har request ka FP16 KV cache ( $p = 2$ ) $= 3.355 \times 1 0^{9}$ bytes (L4.1(b) se, $B = 1$ wali value). $Δ B = ⌊ \frac{19.81 \times 1 0 ^{9}}{3.355 \times 1 0 ^{9}} ⌋ = ⌊ 5.90 ⌋ = 5.$ Toh tum 5 aur requests add kar sakte ho ( $B$ 16 se 21 tak) KV cache ke headroom khaane se pehle. Yahi parent note ki poori kahani hai: weights shrink karo → HBM free karo → bada batch → zyada tokens/sec.

Level 5 — Mastery

Ek open-ended decision problem jo tumhe sahi tool sahi bottleneck ke liye choose karne par majboor karta hai. Intro figure refer karo: tumhara pehla move hamesha yeh poochna hai ki yahan kaunsa bucket (ya padding-idle problem) wall hai?

L5.1 (diagnose and choose) — Do production scenarios. Har ek ke liye, dominant bottleneck identify karo aur {PagedAttention, continuous batching, W4A16 weight-only quantization, W8A8 quantization} mein se ek sabse effective technique choose karo. Formulas ke saath justify karo.

Scenario A: Hazaaron chhote sessions wala ek chatbot (average 60 tokens har ek), lekin server max_len = 2048 KV slots per request reserve karta hai. GPU baar baar OOM kar raha hai bhaale hi average usage tiny hai. Kaunsa bucket overflow ho raha hai, aur kaun si single technique ise fix karti hai?
Scenario B: Ek batch document-summarisation job: bahut bade prompts (hazaaron tokens har ek) bade batch size par process ho rahe hain. Latency matter nahi karti, lekin tum maximum tokens/sec throughput chahte ho, aur model GPU par muskil se fit hota hai. Kaun sa phase dominate karta hai, kaun sa bottleneck follow karta hai, aur kaun si single technique sabse zyada help karti hai?

Recall Solution L5.1

Scenario A → PagedAttention. Diagnose: overflow hone wala bucket teal KV cache hai, aur wajah hai memory waste, weight cost nahi. Har request 2048 slots reserve karti hai lekin 60 use karti hai, apni KV allocation ka $\frac{2048 - 60}{2048} = 97%$ waste karti hai (internal fragmentation). GPU reserved-but-empty KV memory par OOM kar raha hai. Fix: PagedAttention 16-token blocks on demand allocate karta hai, toh ek 60-token session $⌈ 60/16 ⌉ = 4$ blocks $= 64$ slots use karta hai — waste 4 tokens par aajata hai ( $6.25%$ ). Freed memory ⇒ bahut bada batch ⇒ koi OOM nahi. (Continuous batching ek great complement hai lekin root cause yahan fragmentation hai.)

Scenario B → W8A8 quantization. Diagnose: bade batch par bade prompts matlab prefill dominate karta hai, aur prefill compute-bound hai. Weight-only INT4 (W4A16) help nahi karega — yeh compute-bound prefill mein dequant overhead bhi add karta hai (dekho L3.3). Tum yeh bhi chahte ho ki model comfortably fit ho. Fix: W8A8 (8-bit weights aur 8-bit activations) weight bytes half karta hai (fit hone mein madad karta hai) aur matmul INT8 mein chalata hai ek genuine compute speedup ke liye — bilkul wahi jo ek compute-bound prefill workload ko chahiye. Kyunki latency matter nahi karti, tum batch size bhi throughput ke liye badha sakte ho. PagedAttention ko primary lever ke roop mein mat pakdo: prompts lambe hain, toh compute cost ke relative tail-block waste reclaim karne ke liye bahut kam hai.

Master takeaway: pehle poochho "kya yeh memory-bound (decode/KV) hai ya compute-bound (large-batch prefill)?", phir woh lever chuno jo usi wall ko attack karta hai — memory tools (PagedAttention, W4A16) memory walls ke liye, INT8 matmul (W8A8) compute walls ke liye.

Quick self-check

Decode phase ka bottleneck

Memory-bound — single token ke liye saare weights + full KV cache read karta hai.

Worst-case wasted tokens per request PagedAttention mein block_size

B

ke saath

B - 1

(sirf tail block under-full hota hai).

Same value range ke liye INT4 aur INT8 quantization step ka ratio

255/15 \approx 17 \times

bada step INT4 par.

W4A16 ka matlab kya hai

4-bit weights, 16-bit (FP16) activations — weight-only recipe jo memory-bound decode fix karta hai.

Compute-bound large-batch prefill par kaun sa lever attack karta hai

W8A8 (INT8 matmul ek sach mein compute speedup deta hai); weight-only INT4 nahi deta.

Kya INT4 weight-only quantization KV cache shrink karta hai

Nahi — yeh sirf weights shrink karta hai (orange bucket); KV cache (teal) FP16 rehta hai jab tak alag se quantize na karo.