Intuition The one core ideaA neural network stores each of its "weights" as a very precise decimal number, but most of that precision is wasted — so we can replace every weight with the nearest value on a small ruler of evenly spaced marks and save a huge amount of memory. This deep dive builds, from absolute zero, every symbol and picture you need before that sentence makes full sense: what a weight is, what "bits" and "precision" mean, what a number line with marks looks like, and why rounding onto those marks costs almost nothing if you place the marks cleverly.
This is a prerequisites page for the Quantization topic . We assume you have seen nothing . Every symbol used up in the parent note is earned here, in order.
w weight , written w matrix W 7 billion of these numbers w
Picture one weight as a single dot sitting somewhere on a number line.
WHY the topic needs this: quantization does one small thing to every weight — it slides each dot onto the nearest allowed mark. If you don't picture a weight as a point on a line, none of the rounding language later will land. The whole game is: billions of dots, snap each to a nearby mark .
Intuition Why weights cluster near zeroTrained weights are mostly tiny (like 0.03 − 0.11 bell shape hugging 0 — see Weight Distributions in Neural Nets . This shape is why quantization works: we don't need a huge ruler, just a small one centered on 0.
A bit is one yes/no switch: it holds a 0 1 b 2 b 2 b
Count the marks a ruler can have:
bits b
patterns 2 b
name
4
16
INT4
8
256
INT8
16
65 536
FP16
Intuition Picture bits as marks on a rulerMore bits = more marks packed onto the same number line = finer steps between marks = a dot can land closer to where it really wants to be. Fewer bits = fewer marks = each dot must jump further to reach one = more error. That trade — marks vs error — is the entire subject.
WHY the topic needs this: "INT8" and "INT4" are just "how many marks on the ruler." INT8 has 256 marks, INT4 only 16. When the parent says INT4 "hurts," it means 16 marks is a coarse ruler and dots must jump far.
Recall Why does INT4 use so much less memory than FP16?FP16 spends 16 bits per weight; INT4 spends 4. Same weights, one-quarter the bits, so roughly one-quarter the file size — a 4 ×
FP INT float (short for floating-point , e.g. FP16) stores a number with a fractional part, like 0.0374 integer (INT) stores only whole numbers, like − 7 , 0 , 85
Intuition Why swap floats for integers at all?Integers are cheaper to store and cheaper to move through memory, and LLM inference spends most of its time moving numbers , not computing (see Inference Optimization & KV Cache ). Fewer bytes moved → faster. So we want to represent each float weight using an integer index into a ruler . See also Mixed-Precision Training (FP16, BF16) for where FP16 comes from.
We need a rule that turns a float into an integer and back. That rule needs two ingredients, defined next: the scale and the zero-point .
s scale s "how many real units is one step between two neighboring integer marks?" If s = 0.007 14 15 0.007
Read s gap between marks .
s 0.6 integer 85 s = 0.00709 = 0.6027 dictionary that translates integer-language back into real-value-language.
WHY divide by s To turn a real value x "how many steps of size s x — that is division: x / s round to the nearest whole step. This is exactly why the parent's formula opens with x / s
^ x ^ W ^ "the quantized-then-unquantized version" — what you get back after rounding. Comparing x x ^
round ( ⋅ ) round sends a number to the nearest whole integer: round ( 4.67 ) = 5 round ( − 6.5 ) → − 7 − 6
Intuition Where does quantization error come from?A dot at x half a step , i.e. up to s /2 quantization error . Big steps (few bits) → big possible error. This is the picture behind the parent's "error ≈
max error = 2 s , s INT4 = 2 4 ⋅ ... s INT8 — coarser, so larger error.
WHY error jumps when you drop bits: halving the bits does not halve the marks, it square-roots the count... no — it quarters them (256 → 16 doubles s , so the error roughly doubles per bit. That non-linear jump is exactly why the parent warns INT4 needs help (GPTQ, AWQ (Activation-aware Weight Quantization) ).
z zero-point z integer that says "which mark represents the real value 0." It lets us slide the whole ruler left/right so lopsided data (e.g. values from 0 6
Intuition Symmetric vs asymmetric — cover both cases
Symmetric (z = 0 − 8 … 7 weights , whose bell curve is centered on 0.Asymmetric (z = 0 activations after ReLU (all ≥ 0
z Feels right: z z = 4.3 Truth: z a mark , and marks are integers. Forcing z 0 exactly onto a mark with zero error — which matters for padding zeros and ReLU outputs.
[ q m i n , q m a x ] Clamping means: if a computed integer falls off the end of the ruler, pin it to the last mark. For signed INT8 the ends are q m i n = − 128 q m a x = 127 − 8 7
Intuition Why clamping existsA rare huge weight (an outlier ) could compute q = 200 127 finite , so its edges must be enforced. This is the parent's "forgetting to clamp" mistake.
Intuition Why more rulers helpA single wild outlier forces a big max ∣ w ∣ s everyone . Give each small group its own ruler and the outlier only ruins its own tiny neighborhood. More rulers = tighter fit = less error, at the cost of storing more s
The parent's GPTQ section quietly assumes several more symbols. Building them fully belongs to a later deep dive and to Hessian and Second-Order Methods , but here is the minimum vocabulary so nothing is unnamed.
W X ∥ ⋅ ∥ 2
W X calibration set , ~128 samples) fed through the layer to see how it behaves.∥ v ∥ 2 Euclidean length of a list of numbers v v 1 2 + v 2 2 + ⋯ ∥ W X − W ^ X ∥ 2 how much the layer's output changed after quantizing.
H H − 1 H = 2 X X ⊤ second derivatives — it records how sensitive the output error is to nudging each weight, and how weights' effects overlap . Its inverse H − 1 "if I round weight A wrong, how do I nudge weights B, C, … to cancel that mistake." Full story: Hessian and Second-Order Methods .
X W outputs W X see inputs X . That is the whole reason a calibration set exists: to measure output error, not weight error. See also LoRA and QLoRA for how quantized weights combine with trainable adapters.
weight w as a dot on a line
bits give 2 to the b marks
zero point z shifts the ruler
fewer bits means bigger error INT4 hurts
granularity per group rulers
GPTQ needs inputs X and Hessian H
Self-test: cover the right side and answer each aloud before revealing.
A weight w one single number inside the network; picture it as a dot on a number line.
b 2 b
The difference between a float and an integer a float has a fractional part (0.037 − 7
What the scale s the real-value gap between two neighboring integer marks — one "step" of the ruler.
Why we divide by s to count how many steps of size s x x
What the hat in x ^ the reconstructed value s ( q − z ) x
The maximum quantization error about s /2
Why INT4 error is much larger than INT8 16 marks vs 256 → much bigger step s
What the zero-point z names the integer mark that stands for real value 0, letting you shift the ruler for one-sided data.
Why z marks are integers; forcing z
What clamping fixes outliers that compute an integer past the ruler's ends get pinned to q m i n q m a x
Why per-group beats per-tensor a local outlier only stretches its own small ruler, not everyone's, so most weights keep fine steps.
Why GPTQ needs inputs X it minimizes output error ∥ W X − W ^ X ∥ 2