LLM serving (vLLM, quantized inference): Visual walkthrough

5.3.18 · D2 · AI-ML › MLOps & Deployment › LLM serving (vLLM, quantized inference)

Prerequisites jinpar hum rely karte hain (har ek apna vault note hai): Attention Mechanism, GPU Memory & HBM Bandwidth, Batching Strategies, aur parent LLM serving.

Step 1 — Ek number kuch bytes hota hai

KYA. Sabse choti cheez jo ek GPU store karta hai woh hai ek single number — ek weight, ya ek vector ki ek entry. FP16 ("half precision") mein ek number exactly 2 bytes leta hai. Hum ise "bytes per element" kehte hain aur ise naam dete hain $p$ .

YAHAN SE KYUN SHURU KAREIN. Is page ka har memory formula ultimately numbers ki count hai jo bytes per number se multiply hoti hai. Agar hum yeh nail down na karein ki "ek number ki cost kya hai", toh bada formula sirf symbols hai. $p$ woh atom hai.

PICTURE. Figure dekho: red box ek number hai. Woh 2 tiny cells wide hai kyunki FP16 = 2 bytes. Format badlo aur box ki width badal jaati hai — INT8 = 1 cell, INT4 = aadha cell. Woh width hi $p$ hai.

Figure — LLM serving (vLLM, quantized inference)

Step 2 — Ek token ko ek Key vector aur ek Value vector chahiye

KYA. Attention mein, agla token produce karne ke liye model ek query poochta hai aur ise har pehle token ke Key se compare karta hai, phir unke Values ko mix karta hai. Toh har past token ke liye hume do vectors rakhne padte hain: uska Key ( $K$ ) aur uska Value ( $V$ ).

DO KYUN, EK NAHI. Key ka jawaab hai "yeh purana token kitna relevant hai?" aur Value ka jawaab hai "yeh kya content contribute karta hai?". Woh alag-alag roles play karte hain, isliye dono store karne padte hain. Yahi parent ke formula mein mysterious factor 2 ki origin hai — yeh literally "K aur V" hai.

PICTURE. Ek token ek red dot ki tarah khada hai. Usse do stacked bars latkte hain: K-bar aur V-bar. Har bar numbers ki ek list hai (ek vector). Inhe ek baar store karo; kabhi recompute mat karo.

Step 3 — Heads par sum karo: $d_{h e a d}$ ban jaata hai $d_{m o d e l}$

KYA. Attention kai heads mein parallel chalti hai. Agar $n_{h e a d}$ heads hain aur har ek $d_{h e a d}$ length ka vector carry karta hai, toh per token total width hai $n_{h e a d} \cdot d_{h e a d} = d_{m o d e l} .$ Toh heads ko alag track karne ki jagah, hum unhe ek wide number mein collapse karte hain: $d_{m o d e l}$ , model ki "hidden width".

YEH SUBSTITUTION KYUN. Yeh count ko clean banata hai. Parent ne $d_{h e a d}$ nahi balki $d_{m o d e l}$ likha tha exactly isi wajah se — head-vectors ko end-to-end jodte hain toh full hidden dimension wapas ban jaata hai.

PICTURE. Kai short K-bars (ek per head) side by side rakhne par ek lamba bar banta hai jiska length $d_{m o d e l}$ hai. Red brace total width dikhata hai.

Step 4 — Layers stack karo: $L$ se multiply karo

KYA. Ek transformer $L$ layers deep hai, aur har layer apni attention chalati hai, isliye apne K aur V bhi. Toh ek token ka KV footprint $L$ baar repeat hota hai.

ALAG TARIKE SE ADD KYUN NAHI, MULTIPLY KYUN. Har layer Step 3 ki cost ka ek independent copy hai; $L$ identical copies matlab $L$ se multiply karo.

PICTURE. Step 3 ka single-token bar ab $L$ shelves ki ek tall stack mein photocopy ho jaata hai. Red shelf layer 40 hai; count har shelf par same hai.

Step 5 — Kaafi tokens, kaafi users: $S$ aur $B$ se multiply karo

KYA. Ek conversation mein ab tak $S$ tokens hain (sequence length), aur server ek saath $B$ conversations handle karta hai (batch size — Batching Strategies dekho). Har request ka har token Step-4 ki cost pay karta hai.

YEH DO MULTIPLIERS KYUN. KV cache har past token store karta hai (yahi toh poora point hai — recompute nahi), toh length $S$ multiply hoti hai. Aur $B$ independent users matlab $B$ independent caches. Dono multiply karo, phir "numbers ki count" ko "bytes" mein badalne ke liye Step 1 se $p$ lagao.

PICTURE. Ek grid: rows ek conversation ke $S$ tokens hain, aur grid $B$ users ke liye $B$ baar stamp hoti hai. Red column ek user ka growing cache highlight karta hai — har generated token par ek row se lambi hoti jaati hai.

Step 6 — Naive layout iska zyaadaatar hissa waste karta hai

KYA. Purane servers har request ke liye — chahe woh sirf 100 tokens tak pehunche — maximum allowed length ke size ka ek contiguous block reserve karte the, jaise 2048 tokens.

YEH KYUN FAIL HOTA HAI. Used part ( $S$ actual) chhota hai; reserved-lekin-khaali tail badi hai. Woh khaali tail internal fragmentation hai. Aur bhi bura, jab alag-alag lengths ki requests aati-jaati hain, peeche chhodi gayi free memory bikhar jaati hai — external fragmentation — toh naya request ek contiguous slab nahi dhundh sakta chahe total free memory kaafi ho.

PICTURE. max_len tak reserve ek lamba horizontal bar. Sirf left red segment use hota hai; vast black-outlined khaala region waste hai. Neeche, scattered free gaps external fragmentation dikhate hain.

Step 7 — PagedAttention: cache ko blocks mein kaato

KYA. KV cache ko chote fixed-size blocks mein split karo, jaise block_size = 16 tokens. Ek block tab allocate karo jab conversation actually ustak pahunche. Ek block table (ek choti list) record karta hai ki har block physically kahan rehta hai — blocks ka ek doosre ke paas hona zaroori nahi.

YEH WASTE KYUN KHATAM KARTA HAI. Tum un tokens ke liye kabhi reserve nahi karte jo tumhare paas hain hi nahi. Sirf ek last, partly-filled block ka waste hota hai — zyaadah se zyaadah block_size − 1 khaali slots. Non-contiguous blocks external fragmentation bhi theek karte hain: kahin bhi koi bhi free block use kiya ja sakta hai.

PICTURE. Logical token sequence (red, contiguous) upar; block table se arrows use physically alag-alag blocks mein neeche scatter karte hain. Sirf ek single half-empty tail block waste hai.

Step 8 — Edge aur degenerate cases

KYA / KYUN / PICTURE, sab ek figure mein teen panels ke saath:

$S$ exactly block_size ka multiple (jaise $S = 32$ , block 16): last block full hai, waste $= 0$ . Best case — paging free hai.
$S = 1$ (sirf ek token): paging phir bhi ek poora block leta hai, toh waste $= 15$ . Yeh paging ka sabse bura relative case hai, phir bhi naive ke ~2047 ke samne negligible hai.
Shared prefix (copy-on-write): same system prompt wali do requests apne block tables ko same physical blocks ki taraf point karaati hain. Memory ek baar count hoti hai, do baar nahi — ek saving jo naive layout kabhi achieve nahi kar sakta.

Ek picture mein poora summary

Yeh final figure poora safar compress karti hai: ek number → ek token ka K+V → saare heads → saari layers → saare tokens × saare users byte formula build karta hai; phir same bytes reserved-and-wasted (naive) versus paged-and-packed (vLLM) dikhaye jaate hain, freed red memory extra batched users se refill hoti hui.

Recall Feynman retelling — simple words mein bolke batao

Socho ek conversation yaad karna. Har cheez ke liye jo kisi ne bhi kaha, tum do sticky notes rakhte ho: ek jo kitna important tha woh kehti hai (Key) aur ek jo actually kya tha woh kehti hai (Value). Woh hai "2".

Har sticky note numbers ki ek choti list hai jitna model wide hai — woh width, ek baar jab tum saare parallel "heads" ko glue karo, $d_{m o d e l}$ hai. Model conversation ko har layer par dobara padta hai, toh layers $L$ ki sankhya se multiply karo. Tum ab tak har word ke liye ek note rakhte ho — woh $S$ hai — aur tum yeh har user ke liye ek saath kar rahe ho — woh $B$ hai. Aakhir mein har number $p$ bytes cost karta hai. Sab multiply karo: $2 B S L d_{m o d e l} p$ . Ek lambi Llama chat ke liye woh lagbhag 3 gigabytes hai.

Purane servers aisa the jaise har user ke liye ek 2048-slot filing cabinet rent karna chahe unhone sirf 100 notes likhe hon — 95% khaali, aur woh empties scatter hoti hain toh koi aur unhe use nahi kar sakta. PagedAttention iske bajaay on demand chote 16-slot folders deta hai, aur ek table mein likhta hai ki har folder kahan baitha hai. Sirf last half-full folder waste hai. Same 3 GB ab bahut zyaadah conversations hold karta hai, toh GPU ek bada batch serve karta hai — aur zyaadah batch matlab zyaadah tokens per second, kyunki har expensive weight-read ab kaafi users ko feed karta hai.

Recall Quick self-check

KV formula mein factor 2 kahan se aata hai? ::: Ek Key vector aur ek Value vector per token. $d_{h e a d}$ ki jagah $d_{m o d e l}$ kyun? ::: Saare $n_{h e a d}$ heads par sum karne se: $n_{h e a d} \cdot d_{h e a d} = d_{m o d e l}$ . max_len 2048, S 100 ke liye naive worst-case waste? ::: $2048 - 100 = 1948$ tokens. Same request ke liye paged waste (block_size 16)? ::: $⌈ 100/16 ⌉ \cdot 16 - 100 = 12$ tokens. Kya PagedAttention attention math faster banata hai? ::: Nahi — yeh sirf memory layout change karta hai taaki tum zyaadah batch kar sako; FLOPs unchanged rehte hain.