5.3.18 · D1LLM serving (vLLM, quantized inference)
Ek bade language model ko serve karna slow hota hai ek wajah se: har naya word likhne ke liye, machine ko billions of numbers aur har pehle word ki memory dobara padhni padti hai — ek word at a time . Is topic mein sab kuch (KV cache, PagedAttention, quantization) ek trick hai jisse GPU ki memory pipe se kam bytes move hon taaki zyada users ek saath fit ho sakein.
Yeh page kuch bhi assume nahi karta . Agar parent note ne koi symbol, word, ya picture bina explain kiye use kiya, toh hum use pehle yahan banate hain. Isse parent topic se pehle padho.
Ek token text ka ek chunk hota hai — usually ek word ya word ka ek piece — jo ek number mein badal diya jaata hai jise model process kar sake. "unbelievable" 3 tokens ban sakta hai: un, believ, able. Model kabhi letters nahi dekhta; woh ek token-numbers ki sequence dekhta hai.
Ek train sochlo. Har dabba ek token hai . Model train ko left se right padhta hai aur, agla dabba add karne ke liye, use track pe pehle se lage har dabbe ko wapas dekhna hota hai.
Hum in letters ko tokens aur model size count karne ke liye use karenge — picture yaad karo, letter nahi :
Symbol
Plain words
Picture
S sequence length = ab tak kitne tokens hain
train carriages ki ginti
B batch = ek saath kitni alag conversations
parallel tracks pe kitni trains
L model mein layers ki ginti
kitni baar train "thinking machine" se guzarti hai
d m o d e l har token ke vector ki width
ek dabba describe karne ke liye kitne numbers
p bytes per number (FP16 = 2)
ek number store karne ka "wajan"
Ek vector bas numbers ki ek list hoti hai. Model har token ko ek number se nahi, balki ek lambi list se represent karta hai — jaise Llama-2-13B ke liye 5120 numbers. Us length ko d m o d e l
Figure dekho: ek token (ek dabba) d m o d e l numbers ka ek tall stack hai. Jab parent note d m o d e l = 5120
Intuition Length kyun maayane rakhti haiBada d m o d e l har token store karne ke liye zyada bytes bhi. Isliye exactly d m o d e l
Definition Byte aur precisionEk byte 8 bits hota hai — computer memory ki basic unit. Kisi number ki precision woh hoti hai ke hum use store karne mein kitne bytes kharach karte hain:
FP16 (half precision) = 2 bytes per number → p = 2 INT8 = 1 byte → p = 1 INT4 = aadha byte → p = 0.5
Intuition Picture — ek memory pipeSocho ke har number ko storage se compute chip tak ek fixed-width pipe (memory bus) se guzarna padta hai. Ek number mein jitne kam bytes = utne zyada numbers pipe se per second guzar sakte hain. Yahi ek idea quantization ke exist karne ki poori wajah hai — dekho Quantization Fundamentals .
Parent note baar baar Keys , Values , queries kehta rehta hai. Yeh yahan se shuru hota hai, zero se. (Poori detail Attention Mechanism mein hai — hum yahan sirf woh build karte hain jo is topic ko chahiye.)
Definition Query, Key, ValueJab model ek token process karta hai, toh woh us token se teen lists banata hai:
Query (q — "main kya dhundh raha hoon?"Key (k — "main doosron ko jo search kar rahe hain unhe kya offer karta hoon?"Value (v — "actual information jo main match hone par doonga."
Yeh decide karne ke liye ke token t j q t k j v j
Intuition Cache kyun exist karta haiFigure dekho. Token 5 generate karne ke liye, query q 5 k 1 .. k 4 har pehle token ke Values v 1 .. v 4 k v O ( t 2 ) store karte hain. Keys aur Values ka woh stored pile KV cache hai — aur yahi woh memory hog hai jisse poora topic ladta hai.
Definition Attention headModel attention ek baar nahi karta; woh token ke d m o d e l heads kehte hain, har ek independently attend karta hai, phir unhe wapas jointa hai.
n h e a d d h e a d Yeh exactly fit hote hain: n h e a d ⋅ d h e a d = d m o d e l
d m o d e l sabhi heads par storage sum karne se wapas poori width d m o d e l d h e a d d m o d e l
Ek task memory-bound pipe se data move karna ho, aur compute-bound arithmetic karna ho. (Aur zyada GPU Memory & HBM Bandwidth aur Throughput vs Latency Tradeoffs mein.)
Intuition Do phases, pictured
Prefill (poora prompt padhna): saare prompt tokens machine se saath mein ek bade matrix multiply mein guzarte hain → chip busy hai, pipe saath rehti hai → compute-bound .Decode (ek token likhna): sirf ek token ka kaam, lekin phir bhi tumhe saare weights aur pura KV cache pipe se kheenchna padta hai → chip pipe ka wait karta hai → memory-bound .
Yahi is poore topic ka pivot hai: decode memory-bound hai, toh game kam bytes move karna hai.
Quantization ka matlab hai ek fine-grained real number (jo 2 bytes chahta hai) ko chhoti si allowed levels ki set mein se nearest value se replace karna (jo aadha byte chahti hai), taaki store aur move karna sasta ho. Dekho Quantization Fundamentals , GPTQ and AWQ .
Intuition Picture — kam ticks wala ek rulerEk perfect ruler mein infinite marks hote hain. INT4 tumhe sirf 15 marks (2 4 − 1 scale s hai, aur snapping se zyada se zyada aadha gap ka error hota hai. Isliye INT4 chhota aur thoda lossy hota hai — levels kaafi door hain.
s = 2 b − 1 r ma x − r min ( real units per tick, b = bits )
Yahan b bits ki ginti hai, aur 2 b − 1 2 b gaps ki ginti hai. INT4: b = 4 2 4 − 1 = 15
Batching = kaafi conversations ko ek hi pass mein process karna taaki weights (pipe se ek baar padhe jaayein) kaafi users mein reuse hon. Zyada users per weight-read = zyada throughput. Dekho Batching Strategies .
Intuition Yeh memory se kaise link haiBatch mein har conversation ko apna khud ka KV cache chahiye. Toh batch size B GPU pe kitna KV-cache memory fit hota hai . KV memory kam waste karo → bada B
Token = numbered text chunk
Vector of d_model numbers
Query Key Value per token
KV cache stores past K and V
LLM Serving vLLM and Quantized Inference
Arrows ko aise padho: "yeh pehle chahiye woh ke liye." Sab kuch parent topic node T mein funnel hota hai.
A token is ek number jo text ke chunk (word ya word-piece) ko represent karta hai.
A vector is numbers ki ek ordered list; har token d m o d e l
d m o d e l ek token describe karne ke liye kitne numbers use hote hain.
S B L sequence length (tokens), batch (parallel conversations), number of layers.
p bytes per number — FP16 = 2, INT8 = 1, INT4 = 0.5.
Query / Key / Value are per-token lists: main kya dhundh raha hoon / main kya offer karta hoon / woh info jo main hand over karta hoon.
The KV cache exists to har decode step pe past Keys aur Values ko dobara compute karne se bachna (memory ke badle compute trade karta hai).
KV-cache size formula 2 ⋅ B ⋅ S ⋅ L ⋅ d m o d e l ⋅ p
Why d m o d e l d h e a d heads d m o d e l
Memory-bound vs compute-bound slow part data move karna hai vs slow part arithmetic karna hai.
Prefill is poora prompt ek saath padhna — compute-bound.
Decode is ek token at a time likhna — memory-bound.
Quantization is real numbers ko bytes bachane ke liye allowed levels ki chhoti set mein snap karna.
Scale s s = ( r ma x − r min ) / ( 2 b − 1 )
Why batching helps throughput weights ek baar padhe jaate hain aur kaafi conversations mein reuse hote hain.