4.3.13
Intuition Ek-sentence ideaEk neural net apne weights ko high-precision floats (FP16/FP32) mein store karta hai, lekin un bits mein se kaafi redundant precision hoti hai. Quantization un floats ko low-bit integers (INT8, INT4) ke ek chote set pe map kar deta hai, taaki model kam memory use kare aur faster chale — aur ek careful mapping se accuracy almost unchanged rehti hai.
Ek 7B-parameter model FP16 mein sirf weights ke liye 7 × 1 0 9 × 2 bytes = 14 GB 7 × 1 0 9 × 0.5 bytes = 3.5 GB 4× shrink. Yeh farq hai "GPU pe load hi nahi hoga" aur "laptop pe chal jaayega" ke beech. Memory se move hone waale kam bytes → faster inference bhi, kyunki LLM inference memory-bandwidth-bound hoti hai.
KYA problem solve ho rahi hai: floats wasteful hain. Trained nets mein weight distributions roughly bell-shaped aur 0 ke paas clustered hoti hain; "roughly 0.03" kehne ke liye humein 16 bits nahi chahiye.
Definition Affine (asymmetric) quantizationEk real value x b scale s zero-point z q = round ( s x ) + z , x ^ = s ( q − z ) q [ q m i n , q m a x ] x ^ x dequantized approximation hai.
s z [ x m i n , x m a x ] [ q m i n , q m a x ] [ − 128 , 127 ]
Definition Symmetric quantizationAgar hum z = 0 x m a x = − x m i n = max ∣ x ∣ s = q m a x m a x ∣ x ∣ , q = round ( s x ) weights ke liye standard hai, jo 0 ke around near-symmetric hote hain.
Intuition Per-tensor vs per-channel vs per-groupEk single outlier weight max ∣ x ∣ zyaada, chhote scales use karo.
Per-tensor: poori matrix ke liye ek s Per-channel: har output row ke liye ek s Per-group (group size g g s
INT8: 256 levels, signed range [ − 128 , 127 ] INT4: 16 levels, range [ − 8 , 7 ] naive rounding hurt karta hai — accurate rehne ke liye smart methods (GPTQ, AWQ) chahiye. ~4× saving.
Common mistake "Zyaada bits saved hone chahiye toh proportionally zyaada error hona chahiye"Kyun sahi lagta hai: 8→4 se bits halve karna memory halve karta hai, toh surely error ~double ho?
Sachaai: error nonlinearly badhta hai. Har drop hua bit quantization step s , toh rounding error (roughly uniform on [ − s /2 , s /2 ] impact is baat pe depend karta hai ki kaunse weights hain aur tumne compensate kiya ya nahi. Isi liye INT4 ko GPTQ-style compensation chahiye, jabki INT8 ko almost kuch nahi chahiye.
Post-Training Quantization (PTQ): already-trained model ko quantize karo, bina gradient training ke. Fast, cheap. GPTQ/AWQ yahan aate hain.Quantization-Aware Training (QAT): training ke dauran quantization simulate karo taaki model robust banana sikhe. Zyaada accurate, lekin poora training compute chahiye.
LLMs ke liye retraining expensive hai, isliye PTQ dominate karta hai — hence GPTQ.
Intuition Naive rounding dumb kyun haiHar weight ko nearest grid point pe independently round karna yeh ignore karta hai ki weights interact karte hain. Agar tum weight w 1 upar round karte ho, toh tum compensate kar sakte ho abhi-tak unquantized weights ko nudge karke taaki layer ka output sahi rahe. GPTQ exactly yahi karta hai: weights ko ek ek karke quantize karo, aur har ek ke baad, baaki walo ko update karo error absorb karne ke liye.
Definition GPTQ objectiveEk layer ke weight matrix W X output error minimize karo, weight error nahi:
min W ^ ∥ W X − W ^ X ∥ 2 2 Optimal Brain Quantization hai layer-wise apply kiya gaya. Magic quantity hai Hessian:
H = 2 X X ⊤
Intuition GPTQ ka Feynman summaryWeights ko greedily, left to right quantize karo. Jab bhi tum ek weight ko grid pe snap karte ho, ek chhoti si error hoti hai — usi waqt us error ko un neighbors mein spread karo jo abhi untouched hain , is hisaab se ki inputs kitne correlated hain (H − 1 behavior preserve hota hai chahe har weight ab ek 4-bit integer ho.
Worked example 1 — INT8 symmetric quantize ek chhota weight vectorWeights w = [ − 0.9 , 0.1 , 0.6 ] q m a x = 127
max ∣ w ∣ = 0.9 ⇒ s = 0.9/127 = 0.00709 Kyun? Symmetric scale largest magnitude use karta hai.q = round ( w / s ) = [ round ( − 127 ) , round ( 14.1 ) , round ( 84.6 )] = [ − 127 , 14 , 85 ] Dequantize: w ^ = s ⋅ q = [ − 0.900 , 0.0993 , 0.6026 ]
Error: ≈ [ 0 , 0.0007 , 0.0026 ] Chhota kyun? Range [ − 0.9 , 0.9 ] ≈ 0.007
Worked example 2 — Same vector INT4 mein (kyun takleef deta hai)INT4 signed: q m a x = 7 s = 0.9/7 = 0.1286
q = round ( w / s ) = [ − 7 , round ( 0.78 ) , round ( 4.67 )] = [ − 7 , 1 , 5 ] w ^ = s ⋅ q = [ − 0.9 , 0.1286 , 0.6429 ] 0.6 0.043 16× bada . Kyun? Sirf 16 levels matlab step 0.13
Worked example 3 — 7B model ke liye memory math
FP16: 7 B × 2 = 14
INT8: 7 B × 1 = 7
INT4 (+ per-group scales, ~0.5–1 bit overhead): ≈ 3.5 4 Yeh kyun matter karta hai: 6 GB consumer GPU pe fit ho jaata hai. Scale-overhead hi reason hai ki real INT4 files naive 3.5
Common mistake Clamp karna bhool jaanaGalat idea: bas x / s Kyun sahi lagta hai: rounding hi core op hai. Fix: ek outlier q [ q m i n , q m a x ] zaroor clamp karna hai warna integer overflow / garbage milta hai. Clamping hi range ko finite banata hai.
Common mistake Activations ko weights ki tarah quantize karnaGalat idea: sab ke liye symmetric per-tensor use karo. Kyun sahi lagta hai: simpler, uniform pipeline. Fix: activations mein huge outliers hote hain (kuch channels 100× bade). Per-token/per-channel use karo ya outlier channels ko FP16 mein rakho (LLM.int8()/AWQ idea). Weights tame hote hain; activations wild hote hain.
Common mistake Yeh maanna ki GPTQ model ko retrain karta haiGalat idea: GPTQ = 4 bits pe fine-tuning. Fix: GPTQ puri net pe koi backprop nahi karta; yeh ek chhote calibration set (~128 samples) use karke closed-form layerwise least-squares solve karta hai. Yeh PTQ hai, QAT nahi.
Affine quantization mein scale s z q = round ( x / s ) + z s z
Quantization scale s Force karo x m i n → q m i n x m a x → q m a x z s = ( x m a x − x m i n ) / ( q m a x − q m i n )
Weights ke liye symmetric quantization scale? s = max ∣ x ∣/ q m a x z = 0
INT4 INT8 se zyaada mushkil kyun hai? Sirf 16 levels vs 256, toh step s
GPTQ kya minimize karta hai? Layer output error ∥ W X − W ^ X ∥ 2 2
GPTQ mein Hessian kya hai aur kyun? H = 2 X X ⊤
PTQ vs QAT? PTQ ek trained model ko bina training ke quantize karta hai (GPTQ, AWQ); QAT robustness ke liye training ke dauran quantization simulate karta hai.
Per-tensor vs per-group granularity? Per-tensor: poori matrix ke liye ek scale (sasta, worse). Per-group: har block ke liye ek scale (jaise 128 weights) — INT4 ke liye standard, outliers handle karta hai.
7B model ki memory FP16 / INT8 / INT4 mein? 14 GB / 7 GB / ~3.5–4 GB.
Quantization ke dauran clamp kyun karna zaroori hai? Outliers q [ q m i n , q m a x ]
Recall 12-saal ke bachche ko explain karo (Feynman)Socho sabki height ek ruler se describe kar rahe ho. FP16 ek ruler hai jisme hazaar chhote marks hain — super precise lekin carry karna mushkil. Quantization ise sirf 16 marks wale ruler (INT4) se badal deta hai: halka, lekin agar tum sirf har height ko nearest mark pe snap karo, toh kuch logon ki height badly round ho jaati hai. GPTQ woh clever teacher hai jo, ek bachche ki height thodi upar round karne ke baad, agli bacho ko whisper karta hai "thoda chhote khadhe ho" taaki jab tum group add karo , total sahi rahe. Class ek chhoti notebook mein fit ho jaati hai, lekin important totals preserve rehte hain.
Mnemonic Pipeline yaad rakho"Scale, Zero, Round, Clamp, Dequant" → SZ-RCD = "See Zebras Run, Cheetahs Dash." Aur do INT4 helpers ke liye: GPTQ = "Greedily Push The Quantization-error onward."
Mixed-Precision Training (FP16, BF16) — quantization ka training-time cousin.LoRA and QLoRA — QLoRA ek 4-bit (NF4) quantized base model ke upar fine-tune karta hai.Inference Optimization & KV Cache — quantization ek key memory-bandwidth win hai.Hessian and Second-Order Methods — GPTQ ka H = 2 X X ⊤ Weight Distributions in Neural Nets — near-zero, bell-shaped weights quantize kyun well karte hain.AWQ (Activation-aware Weight Quantization) — salient channels protect karta hai; GPTQ ka sibling.
Match xmin xmax to qmin qmax
Symmetric quant for weights
