5.3.18
Intuition Ek line mein core problemLLM serve karna memory-bound hai, compute-bound nahi. Generation ke time pe aap ek token at a time produce karte ho, aur har naye token ke liye aapko saare pichle tokens ko re-attend karna padta hai. Do cheezein cost dominate karti hain:
KV cache — har past token ki stored Keys/Values (sequence length × batch ke saath badhta hai).Weight memory — billions of parameters jo har step pe GPU HBM se read karne padte hain.
vLLM #1 ko PagedAttention low-bit formats (INT8/INT4) mein store karke. Dono milke aapko same GPU pe zyada users serve karne dete hain.
Definition Inference ke do phases
Prefill : poora prompt ek saath process karo → compute-bound, highly parallel (saare prompt tokens ek matmul mein).Decode : output tokens ek-ek karke generate karo → memory-bound, har step mein ek hi token ke kaam ke liye saare weights + poora KV cache read karna padta hai.
KV cache kya hai?
Attention mein, query q t saare pichle tokens 1.. t O ( t 2 ) cache karte hain. Wahi cache memory hog hai.
Worked example Llama-2-13B ke liye KV cache, 1 request, 4096 tokensL = 40 d m o d e l = 5120 p = 2 B = 1 S = 4096 2 ⋅ 1 ⋅ 4096 ⋅ 40 ⋅ 5120 ⋅ 2 = 3.36 × 1 0 9 bytes ≈ 3.1 GB Yeh kyun matter karta hai: sirf ek lambi conversation ~3 GB kha jaati hai. 20 users batch karo → sirf KV cache ka 60 GB. Isliye naive serving itni jaldi OOM ho jaati hai.
Intuition Fragmentation ki problemPurane serving frameworks har request ke liye max_seq_len ke size ka ek contiguous KV memory block reserve karte the. Lekin zyaatar requests chhoti hoti hain. Result:
Internal fragmentation : 2048 slots reserve kiye, 100 use kiye → 95% wasted.External fragmentation : free memory hai lekin ek contiguous chunk mein nahi.
Studies ne paaya ki KV memory ka 60–80% is tarah waste hota tha.
Definition PagedAttention (vLLM ka idea)OS ka virtual memory + paging ka concept udhaar lo. KV cache ko fixed-size blocks block table ke zariye map hoti hai (page table jaisi).
Yeh kyun kaam karta hai: aap tabhi block allocate karte ho jab actually zaroorat ho → near-zero waste (sirf aakhri partial block under-full hota hai). Blocks ko requests ke beech share bhi kiya ja sakta hai (jaise same system prompt) copy-on-write ke zariye.
Throughput kyun jump karta hai: kam wasted memory → bada batch GPU pe fit hota hai → decode step zyada requests per weight-read process karta hai → higher tokens/sec. vLLM naive HF generate ke muqable mein ~24× throughput tak report karta hai.
Definition Continuous (in-flight) batchingPoore batch ke khatam hone ka intezaar karne ki jagah (slowest request sabko rok deta hai), vLLM har step pe finished sequences evict karta hai aur nayi inject karta hai . GPU kabhi padding pe idle nahi rehta. Yeh PagedAttention se alag hai lekin saath mein ship hota hai.
Intuition Quantize kyun karein?Decode is baat se limited hai ki aap HBM se weights kitni jaldi read kar sakte ho . Agar weights FP16 (2 bytes) hain aur aap unhe INT4 (0.5 bytes) banate ho, to aap har step pe 4× kam memory padhte ho → faster decode aur model chhote GPUs pe fit ho jaata hai.
Worked example Ek weight ko INT4 mein quantize karoWeights range [ − 0.6 , 0.9 ] b = 4 = 15 s = ( 0.9 − ( − 0.6 )) /15 = 0.1 z = round ( 0.6/0.1 ) = 6 r = 0.35 q = round ( 0.35/0.1 ) + 6 = 4 + 6 = 10 r ^ = 0.1 ( 10 − 6 ) = 0.4 Error = 0.05 (half a step, expected).
Yeh step kyun? q = 10 s /2
Definition LLM quantization ke flavors
PTQ GPTQ (layer-wise, Hessian use karke output error minimize karta hai), AWQ (activation magnitude se identify kiye gaye salient weight channels ko protect karta hai).QAT (Quantization-Aware Training): training ke dauran quantization simulate karo. Best accuracy, expensive.Weight-only (W4A16): weights INT4, activations FP16 rehti hain — memory-bound decode ko fix karta hai.Weight+activation (W8A8): dono INT8 — INT8 matmul enable karta hai (compute speedup bhi).
Per-tensor : poore tensor ke liye ek ( s , z ) Per-channel / per-group (jaise 128 weights ka group): finer scales → kam error, thoda sa overhead. Modern INT4 (GPTQ/AWQ) group-wise use karta hai.
Common mistake "Quantization hamesha decode speed up karta hai."Kyun sahi lagta hai: kam bytes → padhne ko kam. Catch: W4A16 ko matmul se pehle FP16 mein wapas dequantize karna padta hai (GPUs mein scale pe native INT4 matmul nahi hota). Fayda kam HBM read se aata hai (memory-bound decode), faster arithmetic se nahi . Bade batches ke saath compute-bound prefill ke liye, INT4 weight-only dequant overhead ki wajah se slower bhi ho sakta hai. Fix: jab true compute speedup chahiye to W8A8 use karo.
Common mistake "PagedAttention attention math ko faster banata hai."Kyun sahi lagta hai: iska naam hi PagedAttention hai. Reality: attention computation wahi hai; PagedAttention sirf memory layout/allocation badalta hai taaki aap zyada batch kar sako. Throughput bade batches se badhta hai, faster FLOPs se nahi.
Common mistake "Bada block size hamesha better hota hai."Kyun sahi lagta hai: kam blocks = kam bookkeeping. Reality: bade blocks internal fragmentation wapas laate hain (tail-block waste block_size-1 tokens tak ho sakta hai). Bahut chhota = block-table overhead + poor kernel efficiency. Sweet spot ~16–32.
Common mistake "INT8 INT4 ke muqable mein accuracy loss aadha karta hai, to INT4 do guna bura hai."Kyun sahi lagta hai: linear intuition. Reality: error quantization step s ke saath badhta hai, jo 1/ ( 2 b − 1 ) 15 255 ~17× bada hai, 2× nahi. Isliye 4-bit pe outlier-handling (AWQ/GPTQ, group-wise scales) essential hai.
LLM inference ke do phases kya hain aur kaunsa memory-bound hai? Prefill (prompt process karo, compute-bound) aur Decode (ek token at a time, memory-bound).
KV cache kyun exist karta hai? Har decode step pe saare past tokens ki Keys/Values recompute karne se bachne ke liye; memory ke badle compute trade karta hai.
KV cache size bytes mein formula. 2 ⋅ B ⋅ S ⋅ L ⋅ d m o d e l ⋅ p
PagedAttention kaun sa OS concept use karta hai? Virtual memory / paging — non-contiguous fixed-size blocks jo ek block table ke zariye map hote hain (page table analogy).
Naive KV allocation mein memory waste ke do types? Internal fragmentation (reserved > used) aur external fragmentation (free memory contiguous nahi).
Continuous batching kya hai? Har step pe nayi requests inject karna aur finished ones evict karna taaki GPU kabhi padding pe idle na rahe.
Affine quantization scale formula. s = ( r ma x − r min ) / ( 2 b − 1 )
Zero-point formula. z = round ( − r min / s )
W4A16 ka matlab kya hai? Weights INT4 mein, activations FP16 mein (weight-only quantization).
Weight-only INT4 decode speed up kyun karta hai lekin necessarily prefill nahi? Decode memory-bound hai (chhote HBM read se fayda); prefill compute-bound hai aur dequant-to-FP16 overhead dominate kar sakta hai.
GPTQ aur AWQ mein kya farq hai? GPTQ Hessian info use karke layer output error minimize karta hai; AWQ un salient weight channels ko protect karta hai jo activation magnitude se choose kiye jaate hain.
PagedAttention ke under per-request maximum waste? Zyada se zyada block_size − 1 tokens (partially filled tail block).
INT4 error INT8 se sirf 2× nahi balki bahut zyada kyun hai? Step s ∝ 1/ ( 2 b − 1 )
Recall Feynman: 12 saal ke bachche ko samjhaoSocho AI ek ek word likh kar sentence bana raha hai. Har agla word choose karne ke liye woh conversation ki poori memory dobara padhta hai. Woh memory (the KV cache ) sticky notes ki tarah hai — ek har word ke liye. Purane programs har user ke liye ek bada khaali drawer pakad lete the "bas aise hi," to zyaatar drawers zyaatar khaali rehte the aur desk jaldi bhar jaati thi. vLLM ki jagah chhote barabar boxes use karta hai aur ek tabhi deta hai jab note ko actually ghar chahiye — koi wasted space nahi, to bahut zyada users desk share kar sakte hain. Quantization AI ke dimag (uske numbers) ko shorthand mein likhne jaisi hai: lambe precise numbers ki jagah chhote rounded ones use karo. Thoda kam exact hai lekin unhe bahut jaldi flip kar sakte ho aur zyada le ja sakte ho.
"PAGE your KV, SHRINK your W"
P agedAttention → no fragmentation, A ll blocks small, G reater batch, E vict-and-inject (continuous batching). Phir SHRINK W eights via quantization (S=scale, z=zero-point).
Attention Mechanism — KV cache directly self-attention ke K/V se aata hai.GPU Memory & HBM Bandwidth — kyun decode memory-bound hai.Quantization Fundamentals — affine map, scale/zero-point.GPTQ and AWQ — 4-bit weights ke liye PTQ algorithms.Batching Strategies — static vs continuous/in-flight batching.Model Sharding & Tensor Parallelism — large models fit karne ka complementary tarika.Throughput vs Latency Tradeoffs — kaunse serving metrics optimize karte ho.
Memory-bound not compute-bound
2 · B · S · L · d_model · p
Sequence length and batch
Memory fragmentation 60-80% waste
Fixed-size blocks + block table
