LLM serving (vLLM, quantized inference): Worked examples

5.3.18 · D3 · AI-ML › MLOps & Deployment › LLM serving (vLLM, quantized inference)

Koi bhi number dekhne se pehle, ek reminder of the two formulas jo sab kuch neeche drive karte hain:

Definition Is page par use hone waale conventions (ek baar padh lo)

$b$ = bit-width: bits per stored weight. Yeh integer codes ki count set karta hai $2^{b}$ ; smallest aur largest code ke beech gaps ki count $2^{b} - 1$ hoti hai (isliye woh $s$ mein appear karta hai).
"Levels" is page par hamesha representable ticks ke beech gaps/intervals ki count ko indicate karta hai, $2^{b} - 1$ — woh quantity jo $s$ ke denominator mein hai. Gaps kyun, ticks kyun nahi? Kyunki $s$ ek distance hai (real units per step), aur ek ruler par $n$ ticks sirf $n - 1$ steps chodti hai unke beech. Neeche fence-post figure dekho.
$round (\cdot)$ = round-half-up (ties bade integer ki taraf jaate hain), yahi convention hum throughout use karte hain; alag tie rule ek borderline $q$ ko $1$ se shift kar sakta hai, isliye yeh fix hona chahiye. Neeche tie-breaking figure exactly dikhata hai ki yeh kab bite karta hai.
Memory units: 1 MB $= 102 4^{2}$ bytes (mebibyte), 1 GB $= 102 4^{3}$ bytes (gibibyte). Neeche har "GB" aur "MB" ka matlab binary (GiB/MiB) unit hai jo GPUs report karte hain, jab tak koi step explicitly $1000$ ki power se divide na kare.

Scenario matrix

Har serving/quantization question in cells mein se ek hoti hai. Neeche ke examples [C1]…[C10] label kiye gaye hain taki tum dekh sako ki poora space cover hua hai. Figure same map ek nazar mein deta hai.

Figure — LLM serving (vLLM, quantized inference)

Cell	Case class	Kya toot-ta hai / kya test hota hai
C1	KV cache — small/typical	plug-in sanity, unit tracking
C2	KV cache — batch scaling limit	kab OOM hoga? linear-in- $B$ growth
C3	KV cache — degenerate $S = 1$	1-token prompt ka prefill, tail-block waste
C4	Paged vs naive waste	fragmentation arithmetic, block-size choice
C5	Quantize — interior value	normal rounding, bounded error
C6	Quantize — boundary / clip	$r_{ma x}$ par ya usse upar value saturate hoti hai
C7	Quantize — degenerate range $r_{min} = r_{ma x}$	division by zero, flat-tensor case
C8	Quantize — limit $b \to \infty$	error $\to 0$ , zyada bits nonlinearly kyun help karte hain
C9	Real-world word problem	"kitne users fit honge?" full GPU budget
C10	Exam twist	INT4 vs INT8 error ratio — linear trap se bachna

Worked examples

Worked example

[C1] Ek small model ka KV cache — units sahi karo Ek 6-layer model, $d_{m o d e l} = 512$ , FP16. 1 request of 128 tokens serve karo. KV cache ke kitne megabytes honge?

Forecast: guess karo — kilobytes, ya megabytes? (Yeh log ko surprise karta hai ki ek small model kitna chota hota hai.)

Har symbol identify karo: $B = 1$ , $S = 128$ , $L = 6$ , $d_{m o d e l} = 512$ , $p = 2$ . Yeh step kyun? Yahaan har galti ek wrong symbol hai, wrong math nahi — pehle unhe naam do.
Formula ke according multiply karo: $2 \cdot 1 \cdot 128 \cdot 6 \cdot 512 \cdot 2 = 1, 572, 864$ bytes. Yeh step kyun? Direct plug-in; leading 2 K+V hai, trailing 2 bytes/FP16-element hai — do alag 2s, unhe merge mat karo.
Convert karo: $1, 572, 864/102 4^{2} = 1.5$ MiB. Yeh step kyun? MB yahaan mebibyte ( $102 4^{2}$ bytes) ka matlab hai — woh unit jo GPUs report karte hain; state karo taki "MB" kabhi ambiguous na ho.

Verify: precision INT8 mein halve karo ( $p = 1$ ) → 0.75 MiB ho jaana chahiye. Hota hai. Units: $[—] \cdot [—] \cdot [tok] \cdot [layer] \cdot [val/tok/layer] \cdot [byte/val] =$ bytes. ✓

Worked example

[C2] KV cache — batch scaling jab tak OOM na ho Parent ke Llama-2-13B jaisa hi ( $L = 40$ , $d_{m o d e l} = 5120$ , FP16), $S = 4096$ per request. Ek GPU ke paas KV cache ke liye 40 GiB free hai. Sabse bada batch $B$ kya fit hoga?

Forecast: parent ne kaha tha 1 request ≈ 3.1 GB. Toh… 12? 13?

Per-request KV: $2 \cdot 1 \cdot 4096 \cdot 40 \cdot 5120 \cdot 2 = 3, 355, 443, 200$ bytes. Gibibytes mein: $3, 355, 443, 200/102 4^{3} = 3.125$ GiB. Yeh step kyun? $B$ ek clean linear multiplier hai — ek request ko budget jaisi unit (GiB) mein solve karo, phir divide karo. Note: decimal GB mein ( $/100 0^{3}$ ) yeh $3.355$ GB hai — alag number, isliye hum sab kuch GiB mein lock karte hain.
$B_{ma x} = ⌊ 40/3.125 ⌋ = ⌊ 12.8 ⌋ = 12$ . Yeh step kyun? Tum fractional request fit nahi kar sakte; floor karo, round mat karo.
Leftover: $40 - 12 \times 3.125 = 2.5$ GiB idle — 13th ke liye kaafi nahi. Yeh step kyun? Confirm karta hai ki floor necessary tha aur budget ki "wasted tail" dikhata hai.

Verify: $12 \times 3.125 = 37.5 \leq 40$ ✓ aur $13 \times 3.125 = 40.625 > 40$ ✗. Boundary exactly respected hai. Isliye weights quantize karne se (HBM free karne se) $B$ higher push hota hai.

Worked example

[C3] Degenerate input — 1-token sequence Wahi 13B model, lekin ek request $S = 1$ ke saath (sirf BOS token, generation abhi shuru nahi hui). KV bytes? Aur PagedAttention block_size=16 ke saath, kitna allocate hoga?

Forecast: practically zero cache… lekin ek poora block reserve hoga. Uska kitna waste hoga?

$S = 1$ ke liye KV: $2 \cdot 1 \cdot 1 \cdot 40 \cdot 5120 \cdot 2 = 819, 200$ bytes $\approx 0.78$ MiB. Yeh step kyun? Sanity floor — sabse chhota non-empty case, check karta hai ki formula $S = 1$ par explode nahi karta.
Paging poore blocks allocate karta hai. 1 token ko 16 ka 1 block chahiye → 16 tokens worth reserved: $2 \cdot 16 \cdot 40 \cdot 5120 \cdot 2 = 13, 107, 200$ bytes $\approx 12.5$ MiB. Yeh step kyun? Blocks allocation ki unit hain — tum hardware half-fill nahi kar sakte; yahi tail-block cost hai.
Us block ki wasted fraction: $(16 - 1) /16 = 93.75%$ of one block. Yeh step kyun? Dikhata hai ki worst-case tail waste block_size-1 tokens se bounded hai — exactly parent ka claim, concrete bana diya.

Verify: wasted bytes $= (16 - 1) \times (2 \cdot 40 \cdot 5120 \cdot 2) = 12, 288, 000 \approx 11.7$ MiB, aur $819, 200 + 12, 288, 000 = 13, 107, 200$ = block size. ✓ Parts poore mein add hote hain.

Worked example

[C4] Paged vs naive fragmentation — vLLM ka poora point max_len = 2048, block_size = 16. Ek request actually 100 tokens use karti hai. Naive waste vs paged waste (tokens mein) compare karo.

Figure C4 — Do horizontal bars. Top ("Naive"): ek chhota green "used = 100" segment phir ek huge light-red "wasted = 1948" segment 2048 tak. Bottom ("Paged"): same green 100 plus ek tiny yellow 12-token tail, saath mein 16-token block boundaries mark karne waali blue vertical lines. Visual point: red naive-waste bar yellow paged tail ko dwarf karta hai.

Forecast: naive 2048 reserve karta hai, 100 use karta hai. Paged sirf blocks reserve karta hai. Ratio guess karo.

Naive waste $= max_len - actual = 2048 - 100 = 1948$ tokens. Yeh step kyun? Naive shuru mein contiguous max reserve karta hai — figure mein light-red bar.
Paged: blocks needed $= ⌈ 100/16 ⌉ = 7$ blocks $= 112$ token-slots. Yeh step kyun? Ceiling isliye kyunki 100ven token ko abhi bhi 7th block chahiye (tokens 97–112 rakhta hai).
Paged waste $= 112 - 100 = 12$ tokens (figure mein yellow tail). Yeh step kyun? Sirf final partial block under-full hai — yahi $\leq block_size - 1$ bound hai.

Verify: $12 \leq 16 - 1 = 15$ ✓. Waste reduction factor $= 1948/12 \approx 162 \times$ is request ke liye kam wasted memory. Figure dekho: red naive bar tiny yellow paged tail ko dwarf karta hai.

Worked example

[C5] Ek interior weight ko INT4 mein quantize karo — normal case Weight tensor range $[- 0.6, 0.9]$ , bit-width $b = 4$ . $r = 0.35$ quantize aur dequantize karo.

Figure C5 — −0.6 se 0.9 tak ek real-number line jo yellow ticks se dotted hai (16 representable values, har ek s=0.1 par). Red dot true weight 0.35 mark karta hai; ek red arrow use nearest yellow tick par snap karta hai, blue dot at 0.4 (code q=10). Unka gap, "error = 0.05 = s/2" label kiya gaya, information loss ki puri story hai.

Forecast: hum precision kho denge. Computing se pehle error magnitude guess karo (hint: half a step).

Ticks (representable codes) $= 2^{b} = 16$ ; unke beech gaps $= 2^{4} - 1 = 15$ . Scale $s = (0.9 - (- 0.6)) /15 = 1.5/15 = 0.1$ . Yeh step kyun? $s$ = "real units per gap". Hum gaps ki count (15) se divide karte hain, ticks ki count (16) se nahi — 16 fence-posts ke beech 15 spaces hote hain (fence-post figure yaad karo). $s$ set karta hai ki figure mein grid kitni coarse hai.
Zero-point $z = round (- (- 0.6) /0.1) = round (6) = 6$ . Yeh step kyun? $z$ woh integer hai jo real $0.0$ par map back karta hai; yeh grid align karta hai taki 0 exact tick par land kare (alignment figure).
$q = clip (round (0.35/0.1) + 6, 0, 15) = clip (round (3.5) + 6, 0, 15)$ . Round-half-up ke saath, $round (3.5) = 4$ , toh $q = 4 + 6 = 10$ . Yeh step kyun? Nearest tick par round karo (minimum error); tie rule (half-up) exactly $3.5$ par matter karta hai — yahi us step se $4$ ko $3$ ki jagah choose karwaata hai. Phir $z$ se shift karo; 10 $[0, 15]$ ke andar hai toh koi clip nahi.
Dequantize $\overset{r}{^} = 0.1 (10 - 6) = 0.4$ . Error $= ∣0.4 - 0.35∣ = 0.05$ . Yeh step kyun? $\overset{r}{^}$ woh jagah hai jahan red dot grid par snap hota hai; error gap hai.

Verify: error $0.05 = s /2 = 0.1/2$ ✓ — exactly half-step bound, toh rounding behave kiya. Figure mein 0.35 par red dot nearest yellow tick 0.4 par snap hota hai.

Worked example

[C6] Boundary & clip case — ek outlier weight Same setup ( $s = 0.1$ , $z = 6$ , range $[- 0.6, 0.9]$ ). (a) $r = 0.9$ exactly, aur (b) ek outlier $r = 1.4$ quantize karo.

Forecast: 0.9 range ka top hai — kaun sa integer? Aur 1.4 bahar hai — overflow rokta kya hai?

(a) $q = round (0.9/0.1) + 6 = 9 + 6 = 15$ ; $[0, 15]$ mein hai, koi clip nahi. $\overset{r}{^} = 0.1 (15 - 6) = 0.9$ , error 0. Yeh step kyun? Range endpoints exactly top/bottom tick par land karte hain — design intent; wahan zero error hota hai.
(b) $round (1.4/0.1) + 6 = 14 + 6 = 20$ , jo 15 se zyada hai. Yeh step kyun? Yahi woh overflow hai jiske baare mein parent ne warn kiya tha — ek outlier ek aisi tick chahta hai jo exist nahi karti.
clip $q = 15$ force karta hai. $\overset{r}{^} = 0.1 (15 - 6) = 0.9$ . Error $= ∣0.9 - 1.4∣ = 0.5$ . Yeh step kyun? Wrap-around nahi, saturation: outlier max representable value par cap ho jaata hai.

Verify: clipped error $0.5 ≫ s /2 = 0.05$ ✓ — isliye AWQ/GPTQ outlier channels protect karte hain: ek akela un-clipped-badly weight $10 \times$ normal error carry kar sakta hai. Endpoint 0.9 error 0 deta hai jo confirm karta hai ki grid sahi tarah anchor hua hai. Dekho GPTQ and AWQ aur Quantization Fundamentals.

Worked example

[C7] Degenerate range — ek constant (flat) tensor Ek tensor jahan har weight $0.3$ hai, toh $r_{min} = r_{ma x} = 0.3$ . INT4 mein quantize karo. $s$ ka kya hoga?

Forecast: $s = (0.3 - 0.3) /15 = 0/15 = 0$ … aur phir hum $s$ se divide karte hain. Uh oh.

Naïve scale: $s = (r_{ma x} - r_{min}) /15 = 0$ . Yeh step kyun? Trap expose karo — ek constant tensor ka zero dynamic range hai, toh formula agle step mein zero se divide karta hai.
$q = round (r / s)$ zero se divide karta hai → undefined. Real implementation is guard karta hai. Yeh step kyun? Har quantizer ka yeh edge case hota hai; tumhe isse handle karna hai, crash nahi karna.
Standard fix: jab range $0$ ho, scale ko ek small positive floor par clamp karo $s \leftarrow max (s, ε)$ ek fixed $ε$ ke saath, aur $z = clip (round (- r_{min} / s), 0, 2^{b} - 1)$ set karo. Real frameworks ek tiny default use karte hain (PyTorch/ONNX $ε \approx 2^{- 12} \approx 2.4 \times 1 0^{- 4}$ use karta hai); exact value matter nahi karti kyunki ek constant tensor mein koi information nahi hoti — use exactly reconstruct karne ke liye ek code kaafi hai. Sirf arithmetic ki readability ke liye (realistic default nahi) illustrative value $ε = 0.1$ lo: phir $z = clip (round (- 0.3/0.1), 0, 15) = clip (- 3, 0, 15) = 0$ , $q = clip (round (0.3/0.1) + 0, 0, 15) = 3$ , $\overset{r}{^} = 0.1 (3 - 0) = 0.3$ — exact. Yeh step kyun? $ε$ -floor documented convention hai exactly isliye taki ek flat/dead channel kabhi zero se divide na kare aur apni constant value exactly reconstruct kare. Chosen $ε = 0.1$ hand-calculation ke liye ek round number hai; koi bhi positive floor ek constant ki same exact reconstruction deta hai.

Verify: $ε$ -floor $s = 0.1$ ke saath: $q = 3 \in [0, 15]$ valid hai, zero se divide nahi, aur $\overset{r}{^} = 0.3 = r$ exactly ✓. Realistic $ε = 2^{- 12}$ ke saath re-run karo: $z = clip (round (- 0.3 \cdot 4096), 0, 15) = 0$ , $q = clip (round (0.3 \cdot 4096), 0, 15) = 15$ (clipped), $\overset{r}{^} = 2^{- 12} (15 - 0) \approx 0.00366 \neq = 0.3$ — toh ek tiny $ε$ ek large constant ko reconstruct nahi karta! Honest lesson: production quantizers ek constant channel ko directly value store karke special-case karte hain, $s \to 0$ par trust karke nahi. Hamesha $r_{min} = r_{ma x}$ test karo — pruned/dead channels sach mein flat ho jaate hain.

Worked example

[C8] Limiting behaviour — bits $\to \infty$ hone par kya hota hai Fixed range $[- 1, 1]$ . $b = 4, 8, 16$ ke liye step $s$ aur worst-case error bound $s /2$ compute karo. Limit kya batata hai?

Forecast: zyada bits = kam error. Lekin error kitni tezi se shrink hota hai per bit?

$s (b) = \frac{1 - ( - 1 )}{2 ^{b} - 1} = \frac{2}{2 ^{b} - 1}$ . Worst error $= s /2 = \frac{1}{2 ^{b} - 1}$ . Yeh step kyun? Error bound purely gaps ki count ka function hai ( $2^{b} - 1$ ); zyada gaps, finer grid.
$b = 4 : 1/15 \approx 0.0667$ ; $b = 8 : 1/255 \approx 0.00392$ ; $b = 16 : 1/65535 \approx 1.5 \times 1 0^{- 5}$ . Yeh step kyun? Concrete numbers dikhate hain ki error per bit halve nahi hota — woh roughly $\times 2^{Δ b}$ drop karta hai.
Limit: $b \to \infty$ hone par, $2^{b} - 1 \to \infty$ toh error $\to 0$ — FP16 sirf bahut zyada levels wali quantization hai. Yeh step kyun? FP16 vs INT4 ko same idea different resolutions par frame karta hai; trade-off memory (bytes/element) vs error hai.

Verify: ratio $b = 4$ to $b = 8$ error: $(1/15) / (1/255) = 255/15 = 17$ ✓ — INT4 error INT8 ka ~17× hai, 2× nahi ([C10] ki foreshadowing). Error $b$ mein monotonically decreasing hai ✓.

Worked example

[C9] Real-world word problem — "yeh box kitne users serve kar sakta hai?" Ek A100-80GB (treat karo 80 GiB usable). FP16 Llama-2-13B weights 26 GiB lete hain. Tum INT4 (W4A16) mein quantize karte ho, weights ~6.5 GiB tak shrink hote hain. Activations/overhead ke liye 4 GiB reserve karo. Har user ka KV cache ([C2] se) $S = 4096$ , FP16 par 3.125 GiB hai. Weights quantize karne se kitne zyada concurrent users unlock hote hain?

Forecast: ~19.5 GiB weight memory free ho rahi hai — kitne 3.125 GiB users hote hain yeh?

KV budget, FP16 weights ke saath: $80 - 26 - 4 = 50$ GiB. Users $= ⌊ 50/3.125 ⌋ = ⌊ 16.0 ⌋ = 16$ . Yeh step kyun? Pehle fixed costs (weights + overhead) subtract karo; jo bachta hai woh KV pool hai. Sab terms GiB mein hain toh cleanly add hote hain.
KV budget, INT4 weights ke saath: $80 - 6.5 - 4 = 69.5$ GiB. Users $= ⌊ 69.5/3.125 ⌋ = ⌊ 22.24 ⌋ = 22$ . Yeh step kyun? Same subtraction smaller weight footprint ke saath — freed HBM KV room ban jaata hai.
Extra users unlocked $= 22 - 16 = 6$ . Yeh step kyun? Deliverable — yahaan quantization ka payoff batch size hai, hence throughput (Throughput vs Latency Tradeoffs).

Verify: $16 \times 3.125 = 50.0 \leq 50$ ✓ (exactly full) aur $17 \times 3.125 = 53.125 > 50$ ✗; $22 \times 3.125 = 68.75 \leq 69.5$ ✓ aur $23 \times 3.125 = 71.875 > 69.5$ ✗. Note: win memory-driven hai, arithmetic-driven nahi — parent ka W4A16 dequant mistake box dekho. Related: GPU Memory & HBM Bandwidth, Batching Strategies.

Worked example

[C10] Exam twist — linear trap se bachna "INT8 INT4 se twice the bits use karta hai, toh INT4 ko roughly twice as lossy hona chahiye." Sach ya jhooth, numbers ke saath. (Range $[- 1, 1]$ .)

Forecast: yeh $2 \times$ jaisa lagta hai. Hai kya?

$s$ ka denominator gaps ki count hai $2^{b} - 1$ , bit-width $b$ nahi: INT8 ke $255$ gaps hain, INT4 ke $15$ . Yeh step kyun? Poora trap "bits" (linear, $8$ vs $4$ ) aur "ticks ke beech gaps" (exponential, $2^{b} - 1$ ) ko confuse karna hai.
Error $\propto s \propto 1/ (2^{b} - 1)$ . Ratio $= (2^{8} - 1) / (2^{4} - 1) = 255/15 = 17$ . Yeh step kyun? Error step size se set hota hai, jo bit-width mein exponentially shrink hota hai — toh true factor 17× hai, 2× nahi.
Conclusion: Jhooth. INT4 INT8 se per step ~ $17 \times$ lossier hai, exactly isliye 4-bit ko group-wise scales + outlier protection (GPTQ and AWQ) chahiye usable rehne ke liye. Yeh step kyun? Number ko real engineering se connect karta hai: nonlinearity hi reason hai ki AWQ/GPTQ exist karte hain.

Verify: $255/15 = 17$ ✓ (ek integer, satisfyingly). Sanity: bits double karne se level count roughly square hoti hai ( $2^{8} = (2^{4})^{2} = 256$ ), toh error ~17× drop karna 2× ki jagah consistent hai. ✓

Recall

Recall Har cell, ek line mein

Per-request KV 13B ke liye $S = 4096$ , FP16 par ::: $\approx 3.125$ GiB. 40 GiB KV pool mein us size par sabse bada batch ::: 12 (12.8 ka floor). 100 tokens, max_len 2048, block 16 ke liye Paged vs naive waste ::: 12 tokens vs 1948 tokens (~162× kam). $[- 0.6, 0.9]$ INT4 mein $r = 0.35$ quantize karne se $\overset{r}{^}$ aur error ::: $\overset{r}{^} = 0.4$ , error $0.05 = s /2$ . $r_{min} = r_{ma x}$ case ko kya guard karta hai ::: scale ko ek small positive $ε$ par floor karo (ya constant directly store karo) taki kabhi zero se divide na ho. Usi range ke liye INT4-vs-INT8 ka true error ratio ::: ~17× (2× nahi), kyunki gaps $2^{b} - 1$ hote hain. $s$ kyun $2^{b}$ nahi balki $2^{b} - 1$ se divide karta hai ::: $s$ ek step length hai; $2^{b}$ ticks $2^{b} - 1$ gaps chodti hain (fence-post rule). Zero-point $z$ geometrically kya karta hai ::: integer grid shift karta hai taki tick $q = z$ exactly real value $0.0$ par land kare.