Exercises — Adapter layers and prefix tuning

(a) Adapters. Frozen: all original weights (attention $W_Q,W_K,W_V,W_O$, the FFN, embeddings). Trained: only the adapter's $W_{\text{down}}\in {\mathbb{R}}^{r\times d}$ and $W_{\text{up}}\in {\mathbb{R}}^{d\times r}$ (plus tiny biases). (b) Prefix tuning. Frozen: everything the model shipped with, including $W_Q,W_K,W_V$. Trained: only the prefix matrices $P_k,P_v\in {\mathbb{R}}^{p\times d}$ per layer. Slogan: freeze the giant, train the tiny.

1 → Prefix (we prepend to $K,V$). 2 → Adapter. 3 → Prefix (sequence grows from $n$ to $n+p$). 4 → Adapter ($W_{\text{up}}\to 0$).

WHAT: use $N_{\text{adapter}}\approx 2Lkdr$. WHY: each adapter holds $W_{\text{down}}$ ($r\times d$) and $W_{\text{up}}$ ($d\times r$), each with $dr$ numbers → $2dr$ per adapter; multiply by $Lk$ adapters. $2\cdot 24\cdot 1\cdot 1024\cdot 32=1,572,864\approx 1.57\text{M}.$

$N_{\text{prefix}}=2Lpd=2\cdot 24\cdot 20\cdot 1024=983,040\approx 0.98\text{M}.$ So this prefix ($0.98$M) is a bit cheaper than the adapter of 2.1 ($1.57$M). Notice the swap: $r$ (adapter) plays the same "capacity dial" role that $p$ (prefix) plays — both multiply $2Ld$.

$\frac{1,572,864}{350,000,000}\times 100\approx 0.449\%.$ Well under 1% — the PEFT promise holds.

WHAT: compare $2dr$ (adapter) vs $d^2$ (full map). WHY: the whole point of the bottleneck is $2dr\ll d^2$; find where the inequality flips. $2dr\ge d^2⟺r\ge \frac{d}{2}=384.$ So the adapter is only cheaper while $r<d/2=384$. Beyond that a full map is smaller — which is why we always pick $r\ll d$ (e.g. 16–64). At $r=384$ the two are exactly equal.

Without $\sigma$, $W_{\text{up}}{W}_{\text{down}}$ is a single matrix $M\in {\mathbb{R}}^{d\times d}$. Because it is a product of a $d\times r$ and an $r\times d$ matrix, $\mathrm{rank}(M)\le r$. So the "correction" can only live in an $r$-dimensional subspace — it is a rank-$r$ linear map, incapable of any nonlinear reshaping. Adding $\sigma$ (ReLU/GeLU) breaks the linearity so the two projections no longer collapse, letting the adapter bend the vector nonlinearly. This is the same low-rank story behind LoRA, except LoRA deliberately keeps it linear and adds it to a weight matrix instead.

WHAT: ratio of $(n+p{)}^2$ at the two prefix lengths. $\frac{(512+100{)}^2}{(512+10{)}^2}=\frac{612^2}{522^2}=\frac{374544}{272484}\approx \mathrm{1.375.}$ A ~37.5% increase in attention compute. Lesson: $p$ is a capacity dial, but you pay for it quadratically-ish through the sequence length, so bigger prefixes are not free.

(a) Full FT. One copy $=7\times 10^9\times 4=2.8\times 10^{10}$ bytes $=28$ GB. For 200 tasks: $200\times 28=5600$ GB $\approx 5.6$ TB. (b) PEFT. Backbone once: $28$ GB. Each delta: $1.57\times 10^6\times 4=6.28\times 10^6$ bytes $\approx 6.28$ MB. For 200 deltas: $200\times 6.28=1256$ MB $\approx 1.26$ GB. Total PEFT $=28+1.26\approx 29.26$ GB vs $5600$ GB. Ratio $\approx 5600/29.26\approx 191\times$ less storage. This is the whole business case for PEFT over full FT.

Adapters: $2\cdot 12\cdot 1\cdot 768\cdot 16=294,912$. Prefix: $2\cdot 12\cdot 8\cdot 768=147,456$. Total $=294,912+147,456=442,368\approx 0.442$M. Percentage: $442,368/110,000,000\times 100\approx 0.402\%$. Both mechanisms coexist because they touch different places (adapters after the FFN sublayer via a residual; prefix inside attention K/V) — they don't fight over the same weights.

$r=0$: the bottleneck has zero width — $W_{\text{down}}$ and $W_{\text{up}}$ collapse, the correction term is empty, and $\text{Adapter}(h)=h$ exactly. The adapter is a pure identity: zero new capacity, the model is unchanged. (This is also the initialization target via $W_{\text{up}}\to 0$.) $r=d$: the bottleneck is as wide as the hidden state — no squeeze at all. Params $=2d^2$, twice the cost of a plain full map ($d^2$), and you've lost every efficiency reason to use an adapter. So both extremes are useless; the sweet spot is $0\ll r\ll d$.

At step 0, $W_{\text{up}}\sigma (W_{\text{down}}h)$ is a random nonzero vector added to $h$. So $\text{Adapter}(h)=h+\text{noise}$ — the pretrained forward pass is immediately perturbed at every inserted adapter. Errors compound across $L$ layers, the model's outputs become garbage on step 0, and gradients start from a bad, unfamiliar point (harder optimization, possible forgetting). Near-zero init keeps $\text{Adapter}(h)\approx h$, so training departs from the known-good pretrained behaviour rather than from noise.

The attention output has one row per query. Real tokens are what we want outputs for. If we also prepended $p$ prefix queries, we'd produce $p$ extra output rows for positions that correspond to no real token — meaningless, and they'd mess up the residual stream shape ($n$ in, $n+p$ out). Keeping $Q$ real-only means: real tokens attend over the extra learned $K^{\prime }/V^{\prime }$ slots (soft-mixing in task info via softmax attention) but the number and identity of outputs stays exactly the $n$ real positions. The prefix influences what each real token reads, never how many outputs exist.

$p\to 0$: $K^{\prime }=K$, $V^{\prime }=V$ — the concatenation adds nothing, so the model is exactly the frozen original. Zero steering (mirrors $r\to 0$ for adapters). $p\gg n$: the softmax now competes over $n+p$ keys where $p$ of them are learned. With enough learned slots the attention distribution can be dominated by prefix keys, and real-token context gets drowned out. Plus attention cost is $O((n+p{)}^2)$, so it becomes expensive. The caution: bigger $p$ = more steering and more risk of overriding genuine context, so tune $p$ to task complexity (small $p$ for classification, larger for open generation).

Both methods freeze the backbone. Adapters cost $\approx 2Lkdr$; prefix costs $\approx 2Lpd$. The factor 2 = two learned matrices. $r$ and $p$ are capacity dials with degenerate limits at 0 (no effect) and at their max ($r=d$: no savings; $p\gg n$: drowns context).