Exercises — LoRA and QLoRA

(a) Full update $\Delta W$ is the same shape as $W_0$: $d\cdot k=4096\times 4096=16,777,216\approx 16.7$M. (b) LoRA trains $B$ ($d\times r$) plus $A$ ($r\times k$): $r(d+k)=8(4096+4096)=8\times 8192=65,536$. (c) $\frac{16,777,216}{65,536}=256$. A 256× reduction. WHAT we did: just plugged into "full = $dk$, LoRA = $r(d+k)$". WHY: $B$ and $A$ are the only trainable tensors; the rest is frozen.

Because $B=0$ at the start, $BA=0$, so $\Delta W=0$ and the model behaves exactly like the pretrained one on step 0.

(a) $r(d+k)=8(5120+5120)=8\times 10240=81,920$ per matrix. (b) Two matrices: $2\times 81,920=163,840$. (c) Full for two matrices: $2\times 5120^2=2\times 26,214,400=52,428,800\approx 52.4$M. Ratio $=\frac{163,840}{52,428,800}=0.003125=0.3125\%$ — about 0.3%.

(a) $\frac{\alpha }{r}=\frac{16}{8}=2$. (b) New scale $=\frac{16}{16}=1$. The multiplier dropped from $2$ to $1$, so the effective update is halved — weaker. This is the classic "my rank-16 run felt worse" bug. (c) $\alpha =2r=2\times 16=32$, which restores $\frac{\alpha }{r}=\frac{32}{16}=2$.

(a) $4$ bits $=\frac{4}{8}=0.5$ bytes per parameter. See Quantization of Neural Networks. (b) $7\times 10^9\times 0.5=3.5\times 10^9$ bytes $=3.5$ GB. That is why a 7B QLoRA fine-tune squeezes onto a consumer GPU.

WHAT: the product $BA$ is a sum of $r$ outer products — one per shared index: $BA={\sum }_{i=1}^r{b}_i{a}_i^{\top }$ where $b_i$ is the $i$-th column of $B$ (a $d$-vector) and $a_i^{\top }$ is the $i$-th row of $A$ (a $k$-vector). Each outer product $b_i{a}_i^{\top }$ is a rank-1 matrix (a single "direction"). WHY at most $r$: a sum of $r$ rank-1 matrices has rank $\le r$. The column space of $BA$ is spanned by $\{b_1,\dots ,b_r\}$ — at most $r$ vectors — so $\text{rank}(BA)\le r$. WHAT IT LOOKS LIKE (figure): the full $\Delta W$ is a fat block; $B$ and $A$ are thin strips whose product fills the same block but is constrained to lie in an $r$-dimensional column space. We deliberately throw away the ability to represent higher-rank changes — that is the low-rank bet, discussed in Matrix Rank and Low-Rank Approximation.

Set $r(d+k)=d\cdot k$: $r=\frac{d\cdot k}{d+k}=\frac{1000\times 1000}{1000+1000}=\frac{10^6}{2000}=500.$ For $r\ge 500$ LoRA trains at least as many parameters as full fine-tuning — no saving. Interpretation: LoRA only helps when $r$ is well below this break-even. In practice $r=4$–$16\ll 500$, so the saving is enormous. Beyond that you'd be better off just doing full FT.

1. Base weights — LARGE count, but stored 4-bit → compressed (e.g. 3.5 GB for 7B). Frozen.
2. Optimizer state (Adam: momentum + variance + master copy) — normally the killer at $\sim$12 bytes/param. In QLoRA it exists only for $A,B$, which are tiny → negligible. See Adam Optimizer memory cost.
3. Gradients — also only for $A,B$ → tiny.
4. Activations — depend on batch/sequence length, unrelated to LoRA; handled by Paged Optimizers during spikes. Key point: LoRA kills consumers 2 and 3 (they scale with trainable params, not total params); NF4 shrinks consumer 1.

Budget left for optimizer state: $24-3.5-6=14.5$ GB $=14.5\times 10^9$ bytes. Each trainable param costs $12$ bytes of optimizer state: $N_{\max }=\frac{14.5\times 10^9}{12}\approx 1.208\times 10^9\approx 1.2\text{ billion trainable params}.$ That is astronomically more than any realistic LoRA config needs (typically tens of millions), so you are nowhere near the limit — activations, not adapters, dominate. This is exactly why QLoRA fits a 7B on 24 GB with room to spare.

(a) Merge the chosen adapter into the base: $W\leftarrow W_0+\frac{\alpha }{r}BA.$ Now $W$ is a single ordinary matrix — the forward pass is identical to the base, zero extra latency. This is LoRA's edge over Adapter Layers. (b) You can't bake three different $\frac{\alpha }{r}BA$ corrections into one $W$ simultaneously — merging is per-adapter. So you keep $A,B$ unmerged and add $\frac{\alpha }{r}BAx$ at runtime per request, paying a small extra matmul (two skinny mults). Trade-off: flexibility (swap adapters per request) vs the merged zero-latency win.

Neural-net weights are approximately normally distributed — a bell curve, dense near $0$, sparse in the tails. See the figure.

• Uniform int4 spaces its $16$ levels equally along the axis. Many levels land far out in the tails where almost no weights live — wasted precision — and too few land near $0$ where weights cluster → coarse rounding exactly where it matters.
• NF4 places its $16$ levels at the quantiles of the normal distribution: closely packed near $0$, spread out in the tails. Each bin then holds roughly equal probability mass, which is information-theoretically optimal for Gaussian data. Result: lower average quantization error for the same 4 bits, so the frozen base is a more faithful copy of the original.

(a) $r(d+k)=16(4096+4096)=16\times 8192=131,072$ per projection. (b) $4\times 32\times 131,072=128\times 131,072=16,777,216\approx 16.8$M trainable params. (c) $16,777,216\times 12=201,326,592$ bytes $\approx 0.20$ GB. (d) Base: $7\times 10^9\times 0.5=3.5\times 10^9=3.5$ GB. Total $\approx 3.5+0.20+6=9.7$ GB $<16$ GB. Yes, it fits comfortably, with headroom.

(a) $7\times 10^9\times 12=8.4\times 10^{10}$ bytes $=84$ GB. (This alone won't fit on a single common GPU — the core problem LoRA solves.) (b) From L5.1(c), QLoRA optimizer $\approx 0.20$ GB. Ratio $=\frac{84}{0.201}\approx 418\times$. QLoRA uses roughly 400× less optimizer memory.

Clean start: either choice gives $BA=0$ at step 0 (a zero times anything is zero), so the model starts undisturbed either way. That condition alone doesn't decide it. Gradient flow at step 0 is the deciding factor. The gradient of the loss w.r.t. $B$ depends on $A$; the gradient w.r.t. $A$ depends on $B$ (product rule through $BA$): $\frac{\partial \mathcal{L}}{\partial B}\propto (\text{grad})A^{\top },\frac{\partial \mathcal{L}}{\partial A}\propto B^{\top }(\text{grad}).$

• With $B=0,A=\text{random}$: $\frac{\partial \mathcal{L}}{\partial B}\propto A^{\top }\ne 0$, so $B$ can move immediately; once $B\ne 0$, $A$ also gets gradient. Training unlocks. ✓
• With $A=0,B=\text{random}$: symmetric — $\frac{\partial \mathcal{L}}{\partial A}\propto B^{\top }\ne 0$, so $A$ moves first, then $B$. This also works. Conclusion: both zero-init-one-side schemes start cleanly and unlock. The convention is $B=0$; the essential, non-negotiable requirement is that exactly one of them is zero so $\Delta W=0$ at start while the non-zero side keeps the gradient path alive. Initializing both to zero would freeze training (both gradients zero forever); initializing both random corrupts the output at step 0.

Effective LoRA update strength is controlled by which quantity? ::: The ratio $\frac{\alpha }{r}$, not $\alpha$ or $r$ alone. Full-FT vs QLoRA optimizer memory for 7B (L5.2)? ::: $\approx 84$ GB vs $\approx 0.2$ GB, roughly $400\times$.