Quantization (INT8, INT4, GPTQ): Visual walkthrough

Step 1 — A weight is a point on a ruler; quantizing = snapping to the nearest tick

WHAT. Take one real number, say $w = 0.63$ . A low-bit integer format only allows a handful of evenly spaced values — the grid. Quantizing means moving $w$ to the closest grid tick.

WHY. Everything GPTQ does is built on one atomic act: snap a number to a tick and notice the leftover gap. If we understand that gap, we understand the whole method.

PICTURE. In the figure, the amber dot is the true $w$ . The cyan ticks are the allowed values, spaced by the step $s$ (the distance between two ticks). The white arrow is the jump to the nearest tick $\overset{w}{^}$ .

Figure — Quantization (INT8, INT4, GPTQ)

The naive plan is: snap every weight, throw away every $r$ . The gaps pile up and the layer drifts. GPTQ's whole idea is don't throw $r$ away — spend it.

Step 2 — A layer's output is a weighted sum; that's the thing we must protect

WHAT. A layer does not care about individual weights — it cares about the number it produces. For one output, it multiplies each weight $w_{j}$ by its input $x_{j}$ and adds them: $y = w_{1} x_{1} + w_{2} x_{2} + \dots + w_{n} x_{n} .$

WHY. We are about to change the weights (by quantizing). The reader must see clearly what we are trying to keep unchanged: not the weights, but $y$ . This is the single most important reframing in GPTQ.

PICTURE. Each weight is a lever; each input $x_{j}$ is how hard that lever gets pushed. The output $y$ is the total. Notice the amber lever ( $x$ large): a small wiggle in its weight moves $y$ a lot. The faint lever ( $x$ small): its weight barely matters.

Step 3 — When we snap one weight, the output jumps by a known amount

WHAT. Suppose we quantize just weight $q$ (pick one column). Its residual is $r_{q} = w_{q} - \overset{w}{^}_{q}$ . The output changes by exactly: $Δ y = r_{q} x_{q} .$

WHY. Because $y$ is a plain sum, changing one term by $r_{q} x_{q}$ changes the total by that same amount — nothing else moved yet. This is the "damage" we now want to undo using the weights we haven't quantized.

PICTURE. The blueprint shows $y$ before (cyan bar) and after snapping weight $q$ (amber bar). The gap between the bars is $Δ y = r_{q} x_{q}$ — that gap is what we must erase.

Step 4 — Measuring how weights "talk to each other": the input correlations

WHAT. To cancel $Δ y$ , we adjust the remaining weights. But by how much each? That depends on how the inputs overlap. We collect all the inputs (over many calibration samples) into a matrix $X$ and form: $H = 2 X X^{⊤} .$ Here $H$ (the Hessian) is a table where entry $H_{j k}$ measures how much inputs $j$ and $k$ move together.

WHY THIS TOOL, and why the second derivative. Our error is a quadratic in the weight changes — it's a sum of products of two changes (Step 2 was linear, but the squared output error $∥(\hat{W} - W) X ∥^{2}$ is quadratic). For any quadratic $E (δ) = δ A δ^{⊤}$ , the object that tells you its curvature — how steeply error rises as you move the weights — is its second derivative, and that equals $2 A = 2 X X^{⊤}$ . So the Hessian is not chosen by taste; it is the thing that captures "if I move this weight, how fast does error grow, and in which coupled directions." See Hessian and Second-Order Methods for the general machinery and Weight Distributions in Neural Nets for why calibration inputs matter.

PICTURE. A heat-grid: bright cells = strongly correlated input pairs, dark cells = independent. The diagonal ( $H_{j j}$ ) is each input's own strength. Off-diagonal brightness is what lets one weight substitute for another.

Step 5 — The correction: spread the residual into the untouched weights

WHAT. After locking weight $q$ , GPTQ updates every remaining free weight by: $δ_{rest} = - \frac{w _{q} - w ^ _{q}}{[ H ^{- 1} ] _{q q}} H_{:, q}^{- 1} .$

WHY each piece is there. Read it right-to-left:

$w_{q} - \overset{w}{^}_{q} = r_{q}$ — the residual we must dispose of.
$H_{:, q}^{- 1}$ — the shape of the nudge: it tells each other weight what fraction of the fix to take, based on how it correlates with weight $q$ 's input.
$[H^{- 1}]_{q q}$ in the denominator — a normaliser so the correction is exactly the right size, no more, no less.
Minus sign — we are cancelling, pushing opposite to the error.

PICTURE. Weight $q$ snaps down (amber). The residual, instead of being discarded, flows as cyan arrows into the neighbouring free weights — big arrow to a strongly-correlated neighbour, thin arrow to a weakly-correlated one. The output bar returns to its original height.

Step 6 — Do it left-to-right until every weight is on the grid

WHAT. Repeat Steps 3–5 marching across columns: quantize weight 1, dump its error into weights $2 \dots n$ ; quantize weight 2 (now including the correction it just received), dump its error into $3 \dots n$ ; continue until the last weight — which has nowhere to pass its error, so it just gets snapped.

WHY greedy and ordered. Once a weight is quantized it is frozen; only future (still-free) weights can absorb error. So each step shrinks the pool of helpers by one. Processing in order keeps the bookkeeping a simple triangular sweep — no weight is ever asked to fix something already locked.

PICTURE. A column of weights; a sweep line moves left→right. Behind it: locked amber ticks. Ahead of it: cyan free weights receiving correction arrows from the weight just processed.

Step 7 — The degenerate cases (never leave the reader stranded)

WHAT & WHY & PICTURE, three edge scenarios drawn side by side:

The last weight. No free neighbours remain, so its residual cannot be absorbed — it is simply snapped and its error is the one genuine leftover. (Picture: rightmost tick, no outgoing arrows.)
A dead input, $x_{j} = 0$ . Then column $j$ contributes nothing to $y$ (Step 2 said importance $=$ input size). Its weight can be rounded freely — $H^{- 1}$ handles this because a zero-energy input makes $H_{j j} = 0$ ; GPTQ adds a small damping $λ$ to the diagonal so $H + λ I$ stays invertible.
Perfectly independent inputs ( $H$ diagonal). Then $H^{- 1}$ has no off-diagonal terms, so there is nobody to pass error to — GPTQ gracefully collapses to plain independent rounding (Step 1 for every weight). Compensation only helps when weights are correlated.

The one-picture summary

The whole derivation on one blueprint: a weight snaps to the grid (Step 1) → this jolts the output (Steps 2–3) → the input-correlation table $H$ (Step 4) tells us the cheapest correction → the residual flows into the free neighbours (Step 5) → sweep left-to-right (Step 6) → edge cases handled by damping and the last-weight rule (Step 7). Output preserved; every weight now a low-bit integer.

Recall Feynman retelling — say it back in plain words

I have a row of dials (weights). The machine's answer is the sum of each dial times how hard its knob is pushed (the input). I want to replace every smooth dial with a clicky one that only stops at fixed notches — that saves memory.

If I just clicked each dial to its nearest notch and walked away, all the little clicking errors would add up and the machine would give a wrong answer. So instead I click one dial at a time, and each time I click, I look at the tiny error I made and quietly re-tune the dials I haven't clicked yet so the machine's total answer stays the same. I decide how much each neighbour re-tunes by how similarly their knobs get pushed — that's the correlation table $H$ . I sweep left to right; by the end every dial is clicky, yet the machine still answers exactly as before. The very last dial has no friends left to help it, and any dial with a dead knob gets a free pass. That's GPTQ.

Recall Quick self-test

Why do we minimise output error $∥ W X - \hat{W} X ∥^{2}$ instead of weight error $∥ W - \hat{W} ∥$ ? ::: Because the network only cares about what the layer produces; two weights with the same rounding error hurt the output very differently depending on their input size $x_{j}$ . Why does the Hessian appear as $2 X X^{⊤}$ ? ::: The output error is quadratic in the weight change; the second derivative (curvature) of $δ A δ^{⊤}$ is $2 A$ , and here $A = X X^{⊤}$ . What does $H_{:, q}^{- 1}$ do in the update? ::: It is the shape of the correction — it distributes weight $q$ 's residual across the remaining weights in proportion to how their inputs correlate with $q$ 's.