Quantization (INT8, INT4, GPTQ): Worked examples

The scenario matrix

Every quantization problem you will meet is one of these cells. We will hit all of them.

Cell	Case class	What makes it tricky	Example
A	Symmetric, all-positive values	zero-point unused, range wastes negative half	Ex 1
B	Symmetric, mixed sign	the standard weight case	Ex 2
C	Affine (asymmetric), all-positive (activations after ReLU)	$z \neq = 0$ ; must place 0 exactly	Ex 3
D	Degenerate: all weights equal / $max ∣ x ∣ = 0$	$s = 0$ → division blows up	Ex 4
E	Outlier destroys the range	per-tensor fails, per-group rescues	Ex 5
F	Limiting bit-width: INT8 → INT4 → INT2	error grows as step doubles	Ex 6
G	GPTQ error compensation (interacting weights)	round one weight, fix the rest via $H^{- 1}$	Ex 7
H	Real-world word problem: does it fit the GPU?	bytes, scale overhead, bandwidth	Ex 8
I	Exam twist: clamp an out-of-range value	forgetting the clamp = garbage	Ex 9

Prerequisite ideas we lean on: Weight Distributions in Neural Nets, Hessian and Second-Order Methods, AWQ (Activation-aware Weight Quantization), and the sibling methods in LoRA and QLoRA and Mixed-Precision Training (FP16, BF16).

The picture we keep returning to

Before any numbers: quantization is snapping real values onto a ruler with evenly spaced ticks. The tick spacing is $s$ . A value lands on its nearest tick; the tiny gap between the value and the tick is the error.

Figure — Quantization (INT8, INT4, GPTQ)

Ex 1 — Cell A: symmetric, all positive

Worked example Statement

Weights $w = [0.2, 0.5, 0.8]$ (all positive). Quantize to signed INT8 symmetric, $q_{m a x} = 127$ .

Forecast: guess before reading — will using a signed range on all-positive numbers waste half the ticks?

Step 1. $max ∣ w ∣ = 0.8$ . Why this step? Symmetric scale is set by the largest magnitude so the biggest value reaches the last tick.

Step 2. $s = \frac{0.8}{127} = 0.006299$ . Why? One integer step must cover $\frac{range}{levels}$ ; symmetric uses $q_{m a x}$ as level count.

Step 3. $q = round (w / s) = [round (31.75), round (79.38), round (127.0)] = [32, 79, 127]$ . Why? Rounding is the snap-to-nearest-tick operation.

Step 4. $\overset{w}{^} = s q = [0.20159, 0.49764, 0.8]$ .

Verify: errors $\approx [0.0016, 0.0024, 0]$ — under $s /2 = 0.00315$ everywhere. ✓ The catch (matrix cell A): ticks $- 128 \dots - 1$ (all the negatives) were never used. Half the integer budget wasted. Lesson: for one-signed data, an affine scheme (Ex 3) uses every tick.

Ex 2 — Cell B: symmetric, mixed sign (the normal weight case)

Ex 3 — Cell C: affine (asymmetric) on ReLU activations

Worked example Statement

Post-ReLU activations $a = [0, 1.5, 4.0, 6.0]$ — all $\geq 0$ . Quantize to unsigned INT8, $q_{m i n} = 0, q_{m a x} = 255$ , using the affine formula so we don't waste ticks (fixing Ex 1's flaw).

Forecast: what integer will the real value $0$ map to? (Hint: it is $z$ .)

Step 1. $x_{m i n} = 0, x_{m a x} = 6.0$ . Scale: $s = \frac{x _{m a x} - x _{m i n}}{q _{m a x} - q _{m i n}} = \frac{6.0}{255} = 0.023529$ . Why subtract the endpoints? From the parent derivation, subtracting the two endpoint equations cancels $z$ and isolates $s$ .

Step 2. Zero-point: $z = round (q_{m i n} - \frac{x _{m i n}}{s}) = round (0 - 0) = 0$ . Why? Here $x_{m i n} = 0$ already, so $0$ maps to integer $0$ naturally. (For data starting below 0, $z$ would be positive — that is the whole point of affine.)

Step 3. $q = round (a / s) + z = [0, round (63.75), round (170.0), 255] = [0, 64, 170, 255]$ .

Step 4. $\overset{a}{^} = s (q - z) = [0, 1.50588, 4.0, 6.0]$ .

Verify: every value used a tick from $0$ to $255$ — no wasted budget, unlike Ex 1. Errors under $s /2 = 0.01176$ . ✓

Ex 4 — Cell D: the degenerate zero-range input

Worked example Statement

A tiny quantization group is all identical: $w = [0.3, 0.3, 0.3, 0.3]$ . Symmetric INT4, $q_{m a x} = 7$ . What happens?

Forecast: what is $max ∣ w ∣ - min$ range, and what does that do to $s$ ?

Step 1. For affine here $x_{m a x} - x_{m i n} = 0.3 - 0.3 = 0$ , so the raw formula gives $s = 0/ (q_{m a x} - q_{m i n}) = 0$ . Division by $s$ then explodes. Why does this happen? A ruler with zero tick spacing has no ticks — the map is undefined.

Step 2. The real fix libraries use: guard the scale, $s \leftarrow max (s, ε)$ with a small $ε$ (e.g. $1 0^{- 8}$ ). With symmetric $s = max ∣ w ∣ / q_{m a x} = 0.3/7 = 0.042857$ , which is non-zero, so symmetric mode dodges the trap here.

Step 3. With symmetric $s = 0.042857$ : $q = round (0.3/0.042857) = round (7.0) = 7$ for every entry. $\overset{w}{^} = 7 \times 0.042857 = 0.3$ .

Verify: all four dequantize to exactly $0.3$ , error $= 0$ . ✓ Degenerate group handled: symmetric scale is safe when values share a sign; always clamp $s$ away from $0$ for affine.

Ex 5 — Cell E: one outlier ruins per-tensor, per-group saves it

Worked example Statement

Weights $w = [0.1, - 0.2, 0.15, - 0.05, 8.0]$ . That last value is an outlier. INT4 symmetric ( $q_{m a x} = 7$ ). Compare per-tensor (one $s$ ) vs per-group with two groups $[0.1, - 0.2, 0.15, - 0.05]$ and $[8.0]$ .

Forecast: with one shared scale set by $8.0$ , roughly how many distinct integers will the four small weights collapse onto?

Step 1 (per-tensor). $max ∣ w ∣ = 8.0$ , $s = 8.0/7 = 1.142857$ . Why bad? One giant value stretches the ruler so wide that small weights sit between tick 0 and tick 1.

Step 2. $q_{small} = round ([0.1, - 0.2, 0.15, - 0.05] /1.142857) = [0, 0, 0, 0]$ . All four snap to 0 — total information loss.

Step 3 (per-group). Group 1 alone: $max ∣ \cdot ∣ = 0.2$ , $s_{1} = 0.2/7 = 0.028571$ . $q_{1} = round ([0.1, - 0.2, 0.15, - 0.05] /0.028571) = [4, - 7, 5, - 2]$ . $\overset{w}{^}_{1} = s_{1} q_{1} = [0.114286, - 0.2, 0.142857, - 0.057143]$ .

Verify: per-tensor error on the small weights $\approx [0.1, 0.2, 0.15, 0.05]$ (they became zero). Per-group errors $\approx [0.014, 0, 0.007, 0.007]$ — ~10× smaller. ✓ This is exactly why INT4 uses per-group scales, and why AWQ (Activation-aware Weight Quantization) protects outlier-heavy channels.

Ex 6 — Cell F: the limiting bit-width (INT8 → INT4 → INT2)

Worked example Statement

Fix the value $x = 0.6$ and range $max ∣ x ∣ = 0.9$ . Track how the step $s$ and the rounding error grow as we drop bits: INT8 ( $q_{m a x} = 127$ ), INT4 ( $q_{m a x} = 7$ ), INT2 ( $q_{m a x} = 1$ ).

Forecast: each time we halve the bits, does the step $s$ double, or worse?

Step 1. $s_{8} = 0.9/127 = 0.0070866$ ; $q = round (0.6/ s_{8}) = round (84.67) = 85$ ; $\overset{x}{^}_{8} = 0.602362$ ; error $= 0.002362$ .

Step 2. $s_{4} = 0.9/7 = 0.128571$ ; $q = round (0.6/ s_{4}) = round (4.667) = 5$ ; $\overset{x}{^}_{4} = 0.642857$ ; error $= 0.042857$ .

Step 3. $s_{2} = 0.9/1 = 0.9$ ; $q = round (0.6/ s_{2}) = round (0.667) = 1$ ; $\overset{x}{^}_{2} = 0.9$ ; error $= 0.3$ . Why this step? INT2 has only ${- 1, 0, 1}$ times $s$ ; $0.6$ is nearest to $1 \times 0.9$ .

Verify: step ratios: $s_{4} / s_{8} = 18.1$ , $s_{2} / s_{4} = 7.0$ — the step more than doubles because level count doesn't halve exactly (it's $q_{m a x}$ that changes: 127→7→1). Error jumps $0.0024 \to 0.043 \to 0.3$ . This nonlinear blow-up is the parent note's "more bits ≠ proportional error." ✓

Ex 7 — Cell G: GPTQ error compensation by hand

Worked example Statement

A layer has two weights $w = [w_{1}, w_{2}] = [0.6, 0.6]$ acting on calibration inputs. The (given) inverse Hessian is $H^{- 1} = [1.0 0.8 0.8 1.0] .$ We quantize $w_{1}$ first with INT4 ( $s = 0.9/7 = 0.128571$ ). Show how GPTQ updates $w_{2}$ to absorb the rounding error of $w_{1}$ .

Forecast: after $w_{1}$ rounds up, will the correction push $w_{2}$ up or down?

Step 1. Quantize $w_{1}$ : $q = round (0.6/0.128571) = 5$ , $\overset{w}{^}_{1} = 5 \times 0.128571 = 0.642857$ . The rounding error is $e = \overset{w}{^}_{1} - w_{1} = 0.642857 - 0.6 = 0.042857$ (we overshot). Why compute $e$ ? GPTQ spreads this error, not the raw weight.

Step 2. GPTQ update (parent formula, index $q = 1$ ): $δ_{rest} = - \frac{w ^ _{1} - w _{1}}{[ H ^{- 1} ] _{11}} H_{2, 1}^{- 1} = - \frac{0.042857}{1.0} \times 0.8 = - 0.034286.$ Why divide by $[H^{- 1}]_{11}$ ? It normalises the correction so the output contribution of the fixed weight is matched exactly.

Step 3. Updated $w_{2} = 0.6 + (- 0.034286) = 0.565714$ . Why down? $w_{1}$ overshot (positive $e$ ), and the inputs are positively correlated ( $H_{21}^{- 1} = 0.8 > 0$ ), so $w_{2}$ must drop to keep the layer output constant.

Verify: the correction magnitude $0.034286 = 0.8 \times e$ — exactly the off-diagonal fraction of the error. GPTQ never touches $w_{1}$ again; it now quantizes the updated $w_{2}$ . This is the "greedy, left-to-right, spread the error" loop from the parent note. ✓ See Hessian and Second-Order Methods for why $H^{- 1}$ is the right redistribution weight.

Ex 8 — Cell H: real-world word problem, does it fit?

Worked example Statement

You have a 13B-parameter model and a 12 GB GPU. FP16 won't fit. You quantize weights to INT4 with per-group scales, group size $g = 128$ , and one FP16 scale per group ( $16$ bits). Does it fit? Leave $\sim 2$ GB for the KV cache and activations.

Forecast: guess the INT4 weight size before computing — is it under 10 GB?

Step 1. Raw INT4 weights: $13 \times 1 0^{9} \times 0.5 bytes = 6.5 \times 1 0^{9} bytes = 6.5 GB$ . Why 0.5 bytes? 4 bits $= \frac{1}{2}$ byte per weight.

Step 2. Scale overhead: one 16-bit scale per 128 weights $\Rightarrow \frac{16}{128} = 0.125$ extra bits/weight. Overhead bytes $= 13 \times 1 0^{9} \times 0.125/8 = 0.203 \times 1 0^{9} = 0.203 GB$ . Why include it? Real INT4 files exceed the naive number precisely because of these per-group scales.

Step 3. Total weights $\approx 6.5 + 0.203 = 6.703 GB$ . Add $2$ GB working memory $\Rightarrow 8.703 GB$ .

Verify: $8.703 GB < 12 GB$ → fits, with $\sim 3.3$ GB headroom. ✓ Effective bits/weight $= 4.125$ . This budgeting is what makes quantized inference practical — see Inference Optimization & KV Cache for the KV-cache side.

Ex 9 — Cell I: the exam twist — clamp or die

Worked example Statement

Calibration set $max ∣ w ∣ = 0.5$ gives $s = 0.5/7 = 0.071429$ (INT4, $q_{m a x} = 7$ , $q_{m i n} = - 8$ ). At inference, an unseen weight $w = 0.9$ (larger than calibration saw!) arrives. Quantize it correctly.

Forecast: naive rounding gives what integer — and is it a legal INT4 value?

Step 1 (naive, wrong). $q = round (0.9/0.071429) = round (12.6) = 13$ . Why wrong? INT4 signed only holds $[- 8, 7]$ . Storing $13$ overflows to garbage (e.g. wraps to a negative number in hardware).

Step 2 (correct). Clamp: $q = clamp (13, - 8, 7) = 7$ . Why clamp? Clamping is what makes the range finite; it saturates the outlier at the top tick instead of corrupting memory.

Step 3. $\overset{w}{^} = s q = 0.071429 \times 7 = 0.5$ .

Verify: the outlier saturates at $0.5$ (the calibration max), error $= 0.9 - 0.5 = 0.4$ — large but safe. Without clamping you'd have gotten a nonsensical value. ✓ This is the parent's "forgetting to clamp" mistake, made concrete.

Recap of the matrix

Recall Which example covered which cell?

A → Ex 1 · B → Ex 2 · C → Ex 3 · D → Ex 4 · E → Ex 5 · F → Ex 6 · G → Ex 7 · H → Ex 8 · I → Ex 9

Recall Why does per-group beat per-tensor for outliers?

A single outlier sets the shared $s$ huge, collapsing all small weights to tick 0. A separate scale per small group keeps their ruler fine.

Recall In GPTQ, if a weight rounds up and inputs are positively correlated, which way does the neighbour move?

Down — the correction $- (e / [H^{- 1}]_{q q}) H_{:, q}^{- 1}$ has a negative sign for a positive error and positive off-diagonal.

Cloze checkpoints

The symmetric INT4 scale for $max ∣ w ∣ = 0.9$ is == $0.9/7 = 0.1286$ ==. Dropping from INT8 to INT4 makes the step $s$ grow because ==the level count $q_{m a x}$ shrinks from 127 to 7. Per-group quantization stores one scale per block of $g$ weights to survive outliers. The GPTQ correction pushes error into the remaining weights weighted by the inverse Hessian $H^{- 1}$ . An out-of-range integer must be clamped== to $[q_{m i n}, q_{m a x}]$ before storing.