Exercises — Quantization (INT8, INT4, GPTQ)

The number of levels for $b$ bits is $2^b$.

• INT8: $2^8===256==$ levels, signed range $[-128,127]$.
• INT4: $2^4===16==$ levels, signed range $[-8,7]$.

Why the range isn't symmetric ($-128$ vs $+127$): two's-complement spends one code on $0$, so the negative side gets one extra slot.

• (a)–(ii): $s$ = real units per integer step.
• (b)–(iii): $z$ = integer that real $0$ maps to.
• (c)–(iv): clamp = keep $q$ in the legal range.
• (d)–(i): dequantize = $\hat{x}=s(q-z)$, integer → real.

• GPTQ → PTQ (post-training, no full backprop).
• AWQ → PTQ.
• "simulate rounding during training with backprop" → QAT.

Why symmetric: weights cluster around $0$, so a symmetric grid centered at $0$ wastes no range and needs no zero-point add.

• $\max |w|=0.7\Rightarrow s=0.7/127=0.005512$.
• $q=\text{round}(w/s)$:
  • $-0.4/s=-72.57\to -73$
  • $0.7/s=127\to 127$
  • $0.2/s=36.29\to 36$
  • So $q=[-73,127,36]$.
• $\hat{w}=sq=[-0.4024,0.7000,0.1984]$.
• Error $\hat{w}-w=[-0.0024,0,-0.0016]$.

Why so small: step $s\approx 0.0055$, so any rounding error is at most $s/2\approx 0.0028$.

The tool and why: affine (asymmetric) quantization is used when the data is not centered on $0$ (here it leans positive). We force the two endpoints to line up and solve. $s=\frac{x_{\max }-x_{\min }}{q_{\max }-q_{\min }}=\frac{1.0-(-0.2)}{255-0}=\frac{1.2}{255}=\mathrm{0.004706.}$ $z=\text{round}\left(q_{\min }-\frac{x_{\min }}{s}\right)=\text{round}\left(0-\frac{-0.2}{0.004706}\right)=\text{round}(42.5)=43.$ Why round $z$: the zero-point must be a real integer so that the value $0$ maps exactly to code $43$ (important for padding). Check: does $x_{\max }$ map inside range? $q=\text{round}(1.0/s)+z=213+43=256$ → clamp to 255. This is expected; endpoint rounding can overshoot by one, and clamping fixes it.

• $s=0.7/7=0.1$.
• $q=\text{round}(w/s)=[\text{round}(-4),\text{round}(7),\text{round}(2)]=[-4,7,2]$.
• $\hat{w}=0.1\cdot q=[-0.4,0.7,0.2]$.
• Error on $0.2$: $0$ here — lucky, because $0.2$ is an exact multiple of $s=0.1$.

Lesson: INT4 error is usually large ($s=0.1$ vs $0.0055$ for INT8 → ~18× coarser grid), but if a value lands exactly on a rung the error vanishes. Don't generalize from a lucky vector.

• $\max |w|=3.0\Rightarrow s=3.0/127=0.023622$.
• Quantize $0.015$: $q=\text{round}(0.015/0.023622)=\text{round}(0.635)=1$, so $\hat{w}=0.023622$.
• Error $=0.023622-0.015=0.008622$ — 57% relative error on that weight.

The analysis: the single outlier $3.0$ inflated $\max |w|$ by ~200×, stretching the grid so wide that all the "normal" small weights fall between just a couple of rungs. 253 of the 256 levels sit unused in the empty range between $0.02$ and $3.0$. Fix: per-group scales (one $s$ per block of 128) so the outlier only spoils its own group — see the parent's granularity section.

Why $s^2/12$: for a uniform distribution on width $w$, the variance is $w^2/12$; here width $=s$, giving $s^2/12$.

• INT8: $s_8=0.7/127=0.005512$, ${\sigma }_8=s_8/\sqrt{12}=0.001591$.
• INT4: $s_4=0.7/7=0.1$, ${\sigma }_4=s_4/\sqrt{12}=0.028868$.
• Ratio ${\sigma }_4/{\sigma }_8=s_4/s_8=0.1/0.005512=18.14$.

The point: memory only halved (8→4 bits), but the noise standard deviation jumped ~18×. Each dropped bit doubles $s$, and the step scaling with the number of levels ($2^b$) makes it geometric, not linear. This is precisely why INT4 needs GPTQ/AWQ compensation while INT8 needs almost nothing.

• (a) $w_q-{\hat{w}}_q$ is the rounding error just committed on weight $q$ — the amount we snapped it off the grid.
• (b) $[H^{-1}{]}_{qq}$ is the "self-sensitivity" of weight $q$; dividing normalizes the correction so the pushed error exactly cancels weight $q$'s contribution to the output. It's a Newton-style step (see Hessian and Second-Order Methods).
• (c) The objective minimizes output error $\Vert WX-\hat{W}X{\Vert }^2$. The curvature of that quadratic in the weights depends on how the inputs $X$ correlate: $H=2X{X}^{\top }$. Weights that are hit by correlated inputs must be corrected together — that correlation lives in $X$, not in $W$.

Why add overhead: per-group scales are the reason real INT4 files exceed the naive $3.5$ GB. $\text{bits/weight}=4+\frac{16}{128}=4+0.125=\mathrm{4.125.}$ Bytes per weight $=4.125/8=0.5156$. $\text{size}=7\times 10^9\times 0.5156\text{B}=3.61\text{GB}.$ Interpretation: the extra $0.125$ bit/weight (a 3.1% overhead) buys per-group scales that keep INT4 accurate. Naive $3.5$ GB → real $\approx 3.6$ GB.

The tool: GPTQ tells us output error scales with the input Hessian $H=2X{X}^{\top }$. A larger $H$ (bigger, more energetic inputs) means a given weight rounding error produces a larger output perturbation.

• Layer A: $\sum [H_A{]}_{ii}=1000$ → its weights are hit by high-energy inputs → rounding errors are amplified more.
• Layer B: $\sum [H_B{]}_{ii}=50$ → errors barely propagate. Decision: keep Layer A in INT8 (halve its rounding error there), quantize B hard to INT4. Spend precision where the Hessian says errors are most amplified. This is the seed of mixed-precision quantization; see Mixed-Precision Training (FP16, BF16) and AWQ (Activation-aware Weight Quantization).

Step 1 — quantize $w_1$. Nearest rung to $0.30$ on step $0.5$ is $0.5\cdot \text{round}(0.30/0.5)=0.5\cdot \text{round}(0.6)=0.5\cdot 1=0.5$. So ${\hat{w}}_1=0.5$. Rounding error on weight 1: $w_1-{\hat{w}}_1=0.30-0.50=-0.20$. Step 2 — spread it. The correction to the remaining weights: ${\delta }_{\text{rest}}=-\frac{w_1-{\hat{w}}_1}{[H^{-1}{]}_{11}}{H}_{:,1}^{-1}=-\frac{-0.20}{1}\left[\begin{array}{c}1\\ 0.5\end{array}\right]=\left[\begin{array}{c}0.20\\ 0.10\end{array}\right].$ We only apply the part for the untouched weight $w_2$: ${\delta }_{w_2}=+0.10$. $w_2^{\text{new}}=0.30+0.10=\mathrm{0.40.}$ Result: ${\hat{w}}_1=0.50$, compensated $w_2=0.40$ (to be quantized next). We raised $w_2$ to absorb the fact that $w_1$ was pushed up, keeping the layer's output closer to the original.

Substitute $x_{\min }=-\max |x|$ and $q_{\min }=-q_{\max }$: $s=\frac{x_{\max }-x_{\min }}{q_{\max }-q_{\min }}=\frac{\max |x|-(-\max |x|)}{q_{\max }-(-q_{\max })}=\frac{2\max |x|}{2q_{\max }}=\frac{\max |x|}{q_{\max }}.$ And the zero-point $z=\text{round}(q_{\min }-x_{\min }/s)=\text{round}(-q_{\max }+\max |x|/s)=\text{round}(-q_{\max }+q_{\max })=0$. So symmetric quantization is genuinely the $z=0$ special case. $■$

Effective bits $=4+16/g$:

• $g=32$: $4+0.5=4.5$ bits.
• $g=64$: $4+0.25=4.25$ bits.
• $g=128$: $4+0.125=4.125$ bits.
• $g=256$: $4+0.0625=4.0625$ bits.

Recommendation: $g=128$. Going from 128→64 costs an extra $0.125$ bit/weight (3% more memory) for a marginal accuracy gain, while 128→256 saves only $0.06$ bit but coarsens the scale so outliers spread across twice as many weights. $g=128$ is the community sweet spot precisely because the overhead curve $16/g$ is already nearly flat there.

By definition $\hat{w}=s\cdot \text{round}(w/s)$. Let $r=w/s$. Rounding to the nearest integer means $|\text{round}(r)-r|\le \frac{1}{2}$ (with $\frac{1}{2}$ hit at half-integers). Multiply by $s>0$: $|\hat{w}-w|=s|\text{round}(r)-r|\le s\cdot \frac{1}{2}=\frac{s}{2}.$ Equality is approached when $w$ sits exactly halfway between two rungs, i.e. $w/s$ is a half-integer. This bound is why doubling $s$ (dropping a bit) doubles the worst-case error — the L3.2 nonlinearity in one line. $■$

One line to recall each level. INT4 has how many levels? ::: $2^4=16$. Symmetric scale formula? ::: $s=\max |w|/q_{\max }$, with $z=0$. Why is INT4 error ~18× INT8's at fixed range? ::: step $s$ scales as $1/2^b$, so 4 fewer... no — 4 bits fewer means $2^4$ coarser grid; concretely $s_4/s_8=127/7\approx 18$. What does $H_{:,q}^{-1}$ do in GPTQ? ::: distributes weight $q$'s rounding error into correlated untouched weights. Effective bits for INT4, $g=128$, FP16 scales? ::: $4+16/128=4.125$.