Exercises — Parameter-efficient fine-tuning (PEFT)

• Frozen: $W$ (the giant pretrained weight — never touched).
• Trained: $A$ and $B$ (the two thin low-rank matrices).
• Fixed hyperparameter (not trained): $\alpha$. You choose it; gradient descent does not learn it.

Why: the whole point of PEFT is to leave $W$ alone and only learn a tiny additive correction $BA$. See Low-Rank Matrix Factorization for why $BA$ can be small.

False. Only the frozen base model is loaded in 4-bit. The adapters ($A,B$) stay in higher precision (e.g. 16-bit) so their gradients are meaningful. Quantizing the thing you're learning would destroy the signal you need.

Here $d=4096$ (rows), $k=4096$ (cols).

• (a) $B\in {\mathbb{R}}^{4096\times 8}$, $A\in {\mathbb{R}}^{8\times 4096}$. ($B$ maps the small rank space back up; $A$ squeezes the input down.)
• (b) $r(d+k)=8(4096+4096)=8\times 8192=65,536$.
• (c) $d\cdot k=4096\times 4096=16,777,216$.
• (d) $16,777,216/65,536=256$. 256× fewer parameters.

• Per matrix: $r(d+k)=65,536$ (from 2.1).
• Matrices adapted: $32\text{ layers}\times 2\text{ (}q,v\text{)}=64$.
• Total: $64\times 65,536=4,194,304\approx 4.19\text{M}$.

The $q,v$ choice is standard for Transformer Attention — those projections carry most of the task-relevant adaptation.

• Full: $64\times 4096^2=64\times 16,777,216=1,073,741,824\approx 1.07\text{B}$.
• Fraction trained: $\frac{4,194,304}{1,073,741,824}=\frac{1}{256}\approx 0.39\%$. So LoRA trains about 0.39% of those weights.

Old scale: $\alpha /r=16/8=2$. Keep scale $=2$ at $r=16$: solve $\alpha /16=2\Rightarrow \alpha =32$. Why: the raw product $BA$ tends to grow with $r$; dividing by $r$ decouples "how strong the update is" from "how much capacity it has."

The update term is $\frac{\alpha }{r}BAx$. With $B=0$, the product $BA=0\cdot A=0$ (a zero matrix), so $h=Wx+\frac{\alpha }{r}\underset{=0}{\underbrace{BA}}x=Wx.$ The tuned model starts identical to the pretrained one. Training only adds corrections from zero, so it never blindly overwrites prior knowledge — it drifts away gently. That gradual, additive departure is exactly what mitigates catastrophic forgetting.

Subtle point: you must not zero both $A$ and $B$ — then no gradient could flow and $BA$ would stay $0$ forever. One is zero (output side $B$), one is random (input side $A$), so gradients are nonzero and learning starts.

Look at the figure. An adapter (top) inserts a new sequential block between layers: the data must pass through it in series, so it adds real forward-pass time on every token, forever.

LoRA (bottom) computes $Wx$ and $\frac{\alpha }{r}BAx$ and adds them — a parallel branch, not a series one. After training you fold the branch into a single matrix: $W^{\prime }=W+\frac{\alpha }{r}BA.$ Now inference runs $h=W^{\prime }x$: one matrix multiply, zero extra ops (red arrow). The extra cost existed only during training.

$A\in {\mathbb{R}}^{r\times k}$ has at most $r$ rows, so its row space has dimension $\le r$; hence $\mathrm{rank}(A)\le r$. For any product, $\mathrm{rank}(BA)\le \min (\mathrm{rank}(B),\mathrm{rank}(A))\le r$.

Meaning: LoRA can only produce weight-updates that live in an (at most) $r$-dimensional subspace. If the true task update genuinely needs more directions than $r$, LoRA cannot represent it — you'd need a larger $r$ or full fine-tuning. This is the trade-off knob $r$ controls.

• (a) Full FT: $3\times 14=42\text{ GB}$.
• (b) LoRA: one base + three adapters $=14\text{ GB}+3\times 8\text{ MB}=14,024\text{ MB}$ (using $1\text{ GB}=1000\text{ MB}$).
• Full FT in MB: $42\text{ GB}=42,000\text{ MB}$.
• Ratio: $14,024/42,000=0.3339\dots \approx 33.4\%$. LoRA uses about 33.4% of the storage — and most of that is the single shared base you'd need anyway.
• The marginal cost of each extra task is only 8 MB vs 14 GB — a $14,000/8=1750\times$ saving per new task.

This is the "one model, many cheap plug-ins" idea: hot-swap the 8 MB adapter per request; contrast with Full Fine-Tuning which forces a whole new copy per task.

• 4-bit base footprint: $65\text{B params}\times 0.5\text{ bytes}=32.5\text{ GB}$. This fits in 48 GB with room for activations and the tiny adapters.
• Why it works: the frozen base is only read in the forward pass — 4-bit precision loss there is tolerable. All learning happens in the small 16-bit adapters, so gradients stay precise.
• What flows where: gradients pass through the 4-bit weights (which are dequantized on the fly for the matmul) but only update the 16-bit $A,B$. No optimizer state is stored for the 65B base — only for the ~millions of adapter params.
• Full FT would need $\ge 130\text{ GB}$ (weights) $+\sim 260\text{ GB}$ (Adam state) — impossible on 48 GB. QLoRA's 32.5 GB base makes it feasible.

• Per matrix: $r(d+k)=16(8192+8192)=16\times 16384=262,144$.
• Matrices: $80\times 2=160$.
• Total: $160\times 262,144=41,943,040\approx 41.9\text{M}$.
• Fraction of 65B: $\frac{41,943,040}{65,000,000,000}\approx 0.000645=0.0645\%$. You fine-tune a 65B model by training about 0.06% of its size.

With $r=0$ the matrices $B\in {\mathbb{R}}^{d\times 0}$ and $A\in {\mathbb{R}}^{0\times k}$ are empty; their product is the zero matrix $\Delta W=0$. So $h=Wx$ — the frozen model, unchanged, with nothing trainable. It is the degenerate "no adaptation" baseline: useful only as a control to confirm that any gain you measure truly comes from the LoRA update, not from other training-loop effects.

Two-branch: $h=Wx+\frac{\alpha }{r}(BA)x$. By the distributive law of matrix multiplication over addition, $Wx+\frac{\alpha }{r}(BA)x=(W+\frac{\alpha }{r}BA)x=W^{\prime }x.$ The associativity $\frac{\alpha }{r}(BA)x=\left(\frac{\alpha }{r}BA\right)x$ lets us precompute $\frac{\alpha }{r}BA$ once and fold it into $W^{\prime }$. So merged inference is algebraically identical — that's why it can be lossless and latency-free.

Multi-task hot-swapping. If one server answers requests for summarize, translate, and classify, you keep the single base $W$ in memory and swap the 8 MB adapter per request. If you merged, you'd need three separate full $W^{\prime }$ matrices (each the size of the base) resident — losing the entire storage/memory advantage of Exercise 4.1. So: merge for single-task, low-latency serving; keep unmerged for multi-task hot-swap.

Pick $r=4$. Larger $r$ gives a higher-capacity update that can memorize a tiny dataset (overfit), and it fights LoRA's founding assumption that the true update is low intrinsic dimension. With only 500 examples there's little signal to fill 256 directions — most would fit noise. Empirically returns saturate by $r\approx 8$–$16$; beyond that you spend memory to lose generalization. More rank is not always better.

Trainable params of one LoRA layer? ::: $r(d+k)$. Saving factor for $d=k=4096,r=8$? ::: $256\times$. Why $B=0$ at init? ::: So $BA=0$ and the model starts equal to the pretrained one — no forgetting. Why merged LoRA has zero latency? ::: $W^{\prime }=W+\frac{\alpha }{r}BA$ is one matrix; distributivity makes it exact. When should you NOT merge? ::: Multi-task hot-swap serving — keep the base + swap tiny adapters. 4-bit footprint of a 65B base? ::: $65\text{B}\times 0.5\text{B}=32.5$ GB.