LLM serving (vLLM, quantized inference): Question bank

5.3.18 · D5 · AI-ML › MLOps & Deployment › LLM serving (vLLM, quantized inference)

Definition Is page par use kiye gaye har symbol ki plain words mein explanation

Kisi bhi trap se pehle, poora vocabulary yahan hai — neeche koi bhi symbol aisa nahi jo yahan listed na ho.

Prefill = woh phase jahan model tumhara poora prompt ek saath ek bade parallel matmul (matrix multiplication) mein padhta hai. Bahut saara arithmetic, ek hi shot mein hota hai.
Decode = woh phase jahan model jawab ek token at a time produce karta hai (ek token ≈ ek word-piece). Har step thoda sa kaam karta hai lekin sab kuch re-read karna padta hai.
HBM bandwidth = kitne bytes per second GPU apni fast on-chip memory (HBM = High-Bandwidth Memory) se weights pull kar sakta hai. Dekho GPU Memory & HBM Bandwidth.
KV cache = har past token ki stored Keys ( $K$ ) aur Values ( $V$ ), taaki attention har step unhe recompute na kare.
Continuous batching = scheduler ki yeh aadat ki har single decode step par finished requests ko hatao aur nayi ones ko slot karo, taaki GPU kabhi bhi slowest request ka wait karte hue idle na rahe.
KV-cache formula $2 B S L d_{m o d e l} p$ mein: $B$ = saath mein batched requests ki sankhya, $S$ = sequence length (abhi tak ke tokens), $L$ = transformer layers ki sankhya, $d_{m o d e l}$ = har token ke vector ki width, $p$ = bytes per number (FP16 ke liye 2). Leading $2$ = ek copy $K$ ke liye, ek $V$ ke liye.
$t$ = ek running token index; $O (t^{2})$ matlab hai "cost us square ke saath badhti hai jitne tokens tum dekh chuke ho."
$n_{h e a d}$ = attention heads ki sankhya; $d_{h e a d}$ = ek head ki width. Yeh multiply hokar $d_{m o d e l}$ dete hain.
block_size = ek fixed-size PagedAttention block mein kitne tokens ka KV rehta hai (jaise 16).
Quantization mein: $r$ = ek real weight value, $r_{min}, r_{ma x}$ = range mein sabse chhota/bada real value, $b$ = bits ki sankhya, $q$ = stored integer, $z$ = zero-point (woh integer jo real $0.0$ par decode hota hai), $s$ = step size / scale = do neighbouring integers ke beech real-value gap, $\overset{r}{^}$ = woh value jo rounding ke baad recover hoti hai.

Shuru karne se pehle, teen anchors apne dimaag mein rakho:

Decode memory-bound hai — clock set hoti hai is baat se ki weights + KV cache HBM se kitni tezi se stream hote hain, arithmetic se nahi.
PagedAttention kahan memory rehti hai yeh change karta hai, attention ka math kabhi nahi.
Quantization error step size $s$ se set hoti hai — aur agla callout exactly yeh derive karta hai ki $s$ kyun $1/ (2^{b} - 1)$ jaisi shrink hoti hai.

True or false — justify

PagedAttention attention computation ko khud kam FLOPs mein run karwata hai.

False. Dot-products aur softmax bilkul waisi hi hain; PagedAttention sirf KV memory layout change karta hai taaki bade batches fit ho sakein — throughput batching se badhti hai, faster math se nahi.

Weight-only INT4 (W4A16) hamesha prefill phase ko speed up karta hai.

False. Prefill bade matmuls ke saath compute-bound hota hai; mandatory INT4→FP16 dequant overhead add karta hai jo prefill ko slower bana sakta hai. Win memory-bound decode mein hoti hai.

KV cache sequence length ke saath quadratically badhta hai.

False. Stored cache

S

mein linear hai (

2 B S L d_{m o d e l} p

); quadratic

O (t^{2})

cost woh recomputation hai jo tum caching se bachate ho.

Continuous batching aur PagedAttention ek hi mechanism hain.

False. PagedAttention KV allocation manage karta hai; continuous batching scheduling manage karta hai (finished hatao, har step mein naya inject karo). Yeh saath ship hote hain lekin orthogonal hain.

Symmetric quantization (

z = 0

) strictly worse hai asymmetric se kyunki iska fixed zero-point hota hai.

False. Jo weights roughly zero-centred hain unke liye, symmetric koi range waste nahi karta aur simpler/faster hai; asymmetric tabhi help karta hai jab value range lopsided ho (jaise ReLU activations).

block_size ko jitna bada ho sake banane se bookkeeping minimize hoti hai aur isliye yeh best hai.

False. Bade blocks internal fragmentation phir se introduce karte hain — tail block har request ke liye block_size − 1 tokens tak waste karta hai. Sweet spot ~16–32 hai.

INT4 mein exactly INT8 se do guna accuracy loss hota hai.

False. Error step

s \propto 1/ (2^{b} - 1)

ke saath scale hoti hai: INT8 mein 255 gaps hain, INT4 mein 15, isliye step ~17× bada hai — 2× se kaafi worse.

Weights ko quantize karne se tum inference time par FP16 copy store karna skip kar sakte ho.

True (stored weights ke liye) — tum sirf INT packed weights plus tiny per-group scales rakhte ho; FP16 values har matmul par on the fly reconstruct hoti hain, kabhi persist nahi hoti.

Agar GPU ke paas spare HBM hai, to bade batches hamesha tokens/sec increase karte hain.

False. Sirf tab tak jab tak bandwidth ceiling hit na ho — jab decode HBM saturate kar de, requests add karne se latency badhti hai lekin throughput nahi (Throughput vs Latency Tradeoffs dekho).

Ek hi system prompt share karne wale do requests ko us prompt ke KV ka apna-apna copy zaroor store karna padta hai.

False. PagedAttention unhe copy-on-write ke zariye identical prefix blocks share karne deta hai; private copy tabhi banti hai jab unki generations diverge ho jaayein.

Spot the error

"KV cache size =

B \cdot S \cdot L \cdot d_{m o d e l} \cdot p

Factor 2 missing hai — tum har token ke liye Keys aur Values dono store karte ho, isliye yeh

2 B S L d_{m o d e l} p

hai.

"Hum KV formula mein

d_{h e a d}

use karte hain kyunki attention per head kaam karta hai."

Wrong dimension — tum saare heads par sum karte ho, aur

n_{h e a d} \cdot d_{h e a d} = d_{m o d e l}

, isliye whole-cache formula

d_{m o d e l}

use karta hai.

"AWQ layer ke output error ko Hessian ke saath minimize karke kaam karta hai."

Yeh GPTQ hai. AWQ un salient weight channels ko protect karta hai jo activation magnitude se identify hote hain — koi Hessian nahi.

"W8A8 sirf weights quantize karta hai, isliye yeh W4A16 jaisa hai lekin 8 bits par."

W8A8 activations bhi quantize karta hai, native INT8 matmul enable karta hai (compute speedup); W4A16 activations ko FP16 rakhta hai aur sirf HBM read cut karta hai.

"Zero-point

z = round (r_{min} / s)

Sign error — yeh

z = round (- r_{min} / s)

hai, woh integer jo

\overset{r}{^} = s (q - z)

ke under real

0.0

par map karta hai.

"Per-tensor quantization per-channel se finer-grained hai."

Ulta hai. Per-tensor ek

(s, z)

use karta hai sab kuch ke liye (coarsest); per-channel/per-group kaafi saare scales use karta hai (finer, lower error).

"PagedAttention saara KV memory waste eliminate kar deta hai."

Sab nahi — tail block abhi bhi partly empty hota hai, isliye waste block_size − 1 tokens per request se bounded hai, sirf naive contiguous reservation se kaafi chhota.

"Naive framework memory fragment karta hai kyunki attention

O (t^{2})

hai."

Unrelated causes. Fragmentation is wajah se aata hai ki har request ke liye max_seq_len contiguous blocks reserve hote hain, attention cost se nahi.

Why questions

Decode memory-bound kyun hai jabki prefill compute-bound hai?

Prefill saare prompt tokens ko ek bade parallel matmul mein run karta hai (har weight-read par bahut saara arithmetic); decode ek token ka kaam karta hai jabki saare weights + KV padhta hai, isliye bandwidth dominate karta hai.

Weights ko INT4 tak reduce karna decode speed up kyun karta hai jabhi matmul abhi bhi FP16 mein run karta hai?

Bottleneck HBM se weights padhna hai; INT4 fetch karne ke liye 4× kam bytes hain, aur dequant us saved transfer ke relative cheap hai — arithmetic limiter nahi hai.

Quantization rounding aur clipping dono use kyun karta hai sirf ek ke bajaye?

Rounding ek real value ko nearest representable integer par map karta hai (error

s /2

se bound karta hai); clipping out-of-range outliers ko extremes par saturate karta hai wrapping/overflowing ke bajaye.

4-bit methods group-wise scales aur outlier handling par itna heavily kyun rely karte hain?

Sirf 15 gaps ke saath step

s

bada hota hai, isliye ek per-tensor scale jo ek outlier se bloat ho, baaki har weight ko barbad kar deta hai; per-group scales step tight rakhte hain jahaan values actually cluster hoti hain.

PagedAttention throughput ~24× raise kar sakta hai lekin single request ki latency ko touch kyun nahi karta?

Yeh zyada requests same HBM mein pack karta hai (bada batch → har weight-read par zyada useful kaam), lekin ek request ka apna token-by-token generation path unchanged rehta hai.

Tensor parallelism decode ki memory-bound nature kyun nahi hatata?

Sharding weights aur KV ko GPUs mein split karta hai bade models fit karne ke liye, lekin har GPU abhi bhi har step mein apna shard HBM se stream karta hai plus cross-GPU communication add hoti hai — per-step bottleneck bandwidth rehta hai.

Edge cases

Us request ke liye KV waste kya hai jiska length block_size ka exact multiple hai?

Zero tail waste — har block completely full hai, PagedAttention ke liye best case.

Us weight ke liye quantization error kya hoti hai jo exactly do levels ke beech halfway land kare?

Yeh maximum per-element error $s /2$ hit karta hai; rounding ko ek side choose karni padti hai, aur half a step worst case hai.

r_{ma x}

se far upar ek real value ke liye quantization output kya hoga?

Yeh top integer

2^{b} - 1

par clip ho jaata hai; dequantizing

\overset{r}{^} = r_{ma x}

deta hai (outlier overflow ke bajaye saturate ho jaata hai).

r_{min}

se far neeche ek real value ke liye quantization output kya hoga?

Yeh bottom integer

0

par clip ho jaata hai; dequantizing

\overset{r}{^} = s (0 - z) = r_{min}

deta hai, lower bound par mirror-image saturation.

Batch size 1 aur short prompt ke saath, kya PagedAttention throughput mein help karta hai?

Barely — iska win kaafi saari requests pack karne se aata hai; ek chhoti request ke saath reclaim karne ke liye thoda fragmentation hota hai, isliye gains minimal hain.

Agar prefix share karne wale do requests ek shared block mein naya token likhne ki koshish karein to kya hoga?

Copy-on-write trigger hota hai — shared block duplicate ho jaata hai taaki har request apni copy mein likh sake, isolation preserve hoti hai.

Agar block_size = 1 ho, to tumne effectively kya banaya hai?

Per-token allocation zero tail waste ke saath lekin maximal block-table overhead aur poor kernel efficiency — bade block se bilkul opposite extreme.

Jab generation bilkul pehle decode step (

S = 1

) par ho to KV cache size kya hai?

Sirf ek token ka:

2 B L d_{m o d e l} p

— cache chhota shuru hota hai aur tokens accumulate hone par linearly badhta hai.

Recall Ek-sentence self-test

Answers cover karo aur explain karo: quantization decode par kyun jeetata hai lekin prefill par risk kyun leta hai? Answer ::: Decode memory-bound hai isliye kam weight bytes = faster; prefill compute-bound hai isliye extra dequant work chhote read ko outweigh kar sakta hai.