Adapter layers and prefix tuning: Worked examples

Symbols we reuse (all defined in the parent, restated so line one stands alone):

$d$ = hidden dimension (width of a token's vector). Picture a column of $d$ numbers.
$r$ = adapter bottleneck dimension, the squeezed middle width. $r ≪ d$ means "much smaller".
$L$ = number of transformer layers stacked top to bottom.
$k$ = adapters inserted per layer.
$p$ = prefix length = number of learned fake tokens prepended.
$n$ = real sequence length (how many actual words).
Adapter params $\approx 2 L k d r$ · Prefix params $= 2 L p d$ .

The scenario matrix

Cell	Case class	Covered by
C1	Adapter, normal regime $r ≪ d$	Ex 1
C2	Adapter, degenerate $r = 0$	Ex 2
C3	Adapter, saturated $r = d$ (no saving)	Ex 3
C4	Adapter, $σ$ removed (rank collapse)	Ex 4
C5	Prefix, normal small $p$	Ex 5
C6	Prefix, degenerate $p = 0$	Ex 5
C7	Prefix, limiting large $p$ (compute blow-up)	Ex 6
C8	Head-to-head: pick cheaper method	Ex 7
C9	Real-world serving-cost word problem	Ex 8
C10	Exam twist: init-to-identity failure	Ex 9

Each example below is tagged with the cell(s) it fills.

Ex 1 — C1: adapter in the normal regime A model has $d = 1024$ , $L = 24$ layers, $k = 2$ adapters per layer, bottleneck $r = 16$ . The frozen backbone has $350$ M params. (a) Params per adapter (ignore biases)? (b) Total trainable? (c) As a percentage of the backbone?

Forecast: guess — is it above or below 1%?

Per-adapter weights $= 2 d r = 2 \cdot 1024 \cdot 16 = 32, 768$ . Why this step? An adapter is down-project ( $d \to r$ , that's $d r$ numbers) then up-project ( $r \to d$ , another $d r$ ). Two matrices → $2 d r$ .
Total $= 2 L k d r = 2 \cdot 24 \cdot 2 \cdot 1024 \cdot 16 = 1, 572, 864 \approx 1.57 M$ . Why? Multiply per-adapter by how many adapters exist: $L$ layers $\times$ $k$ each.
Percentage $= \frac{1.5729 M}{350 M} \times 100% \approx 0.449%$ . Why? This is the whole point of PEFT — we quantify "tiny".

Verify: $1.57 M ≪ 350 M$ , ratio under 1% — matches the "0.1–3%" band the parent promised. Units: pure count / count = dimensionless %. ✓

Worked example

Ex 2 — C2: the degenerate $r = 0$ What transformation does an adapter compute if we set $r = 0$ ? What does the module output?

Forecast: does anything survive?

Count params: $2 d r = 2 d \cdot 0 = 0$ . No trainable weights. Why this step? A bottleneck of width zero has no numbers in it.
The down-projection $W_{down} \in R^{0 \times d}$ maps every vector to the empty vector; up-projection $W_{up} \in R^{d \times 0}$ can only output the zero vector. Why? A matrix with zero rows/columns produces the zero-length / zero result.
So $Adapter (h) = h + 0 = h$ . The module is the exact identity, forever. Why it matters? $r = 0$ = "no adapter". It's the floor of capacity — the frozen model passes through untouched.

Verify: matches the residual init trick: $W_{up} \to 0$ gives near-identity; $r = 0$ gives exact identity with zero cost. Consistent. ✓

Worked example

Ex 4 — C4: remove the nonlinearity (rank collapse) Drop $σ$ so the adapter is $h + W_{up} W_{down} h$ . With $d = 1024, r = 16$ , what is the rank of the correction it can apply, and why is that a limitation?

Forecast: how expressive is a linear-only bottleneck?

Collapse the two matrices: $W_{up} W_{down} = M$ , a single $d \times d$ matrix. Why this step? Matrix times matrix is one matrix — no $σ$ to keep them separate.
Rank bound: $rank (M) \leq min (d, r) = min (1024, 16) = 16$ . Why? Multiplying by a width- $r$ waist forces the output through 16 dimensions; the correction lives in a 16-D subspace of a 1024-D space.
Consequence: the adapter can only add a rank-16 linear correction — no curved (nonlinear) reshaping. Inserting $σ$ breaks the collapse, letting the two matrices act independently and produce nonlinear corrections. Why it matters? This is exactly the linear-vs-nonlinear boundary; compare to LoRA and low-rank adaptation, which intentionally stays linear and low-rank.

Verify: $min (1024, 16) = 16$ , so without $σ$ we lose to a rank-16 linear map — matches parent step 2 "two stacked linears collapse to one". ✓

Prefix examples

Worked example

Ex 6 — C7: large $p$ and the compute blow-up Attention cost scales as $O ((n + p)^{2})$ per layer. For a real sequence $n = 50$ , compare the per-layer attention work at $p = 10$ vs $p = 200$ .

Forecast: does big $p$ cost you linearly or worse?

$p = 10$ : effective length $n + p = 60$ , work $\propto 6 0^{2} = 3600$ . Why this step? Every query attends over $n + p$ keys, and there are $n$ queries → the $(n + p)^{2}$ scaling dominates.
$p = 200$ : effective length $250$ , work $\propto 25 0^{2} = 62, 500$ . Why? Same formula, bigger prefix.
Ratio $= 62, 500/3600 \approx 17.36 \times$ more attention work, while param count only grew $20 \times$ linearly. Why it matters? $p$ is a capacity dial (params grow linearly), but the compute per token grows quadratically once $p$ dominates $n$ . Long prefixes on short inputs are expensive.

Verify: $6 0^{2} = 3600$ , $25 0^{2} = 62500$ , ratio $\approx 17.4$ . Quadratic scaling confirmed; contrast with Prompt tuning and soft prompts where the prompt is even smaller. ✓

Word problem & exam twist

Worked example

Ex 8 — C9: serving 100 customers (real-world cost) A company serves 100 customers, each with a custom fine-tune of a 7B-param model stored in fp32 (4 bytes/param). Compare disk for full fine-tuning vs PEFT where each customer's delta is $2 M$ params, backbone shared once.

Forecast: rough order of magnitude of the saving?

Full FT storage $= 100 \times 7 \times 1 0^{9} \times 4$ bytes $= 2.8 \times 1 0^{12}$ bytes $= 2800 GB$ . Why this step? Full FT = one whole 28 GB copy per customer, times 100.
PEFT storage $=$ one backbone $(7 \times 1 0^{9} \times 4 = 28 GB)$ $+$ $100 \times (2 \times 1 0^{6} \times 4 = 8 MB) = 28 GB + 800 MB = 28.8 GB$ . Why? The 28 GB brain is shared; only the tiny 8 MB deltas multiply.
Saving factor $= 2800/28.8 \approx 97.2 \times$ less disk. Why it matters? This is the storage argument from the parent's "storage/compute problem", made concrete. It also dodges Catastrophic forgetting since the frozen backbone never drifts.

Verify: $28 GB = 28, 000 MB$ ; $+ 800 MB = 28, 800 MB$ . $2, 800, 000 MB /28, 800 MB \approx 97.2$ . Two orders of magnitude — matches "100 copies vs one" intuition. ✓

Worked example

Ex 9 — C10: exam twist — the init-to-identity trap An engineer initializes an adapter's $W_{up}$ randomly (like a normal layer) with entries of scale $\approx 0.02$ , and $W_{down}$ likewise. At step 0, the residual output is $h + W_{up} σ (W_{down} h)$ . Explain, with a picture, why the frozen model's output is corrupted, and what fixes it.

Forecast: does the pretrained forward pass survive step 0?

Trace the signal: $W_{down} h$ is a nonzero random vector, $σ (\cdot)$ passes a nonzero part, $W_{up}$ maps it to a nonzero perturbation $δ$ . So output $= h + δ$ , not $h$ . Why this step? Random weights inject noise into every hidden state on step 0 — the pretrained activations shift before any learning.
See it: the figure below shows the clean pretrained vector $h$ (black) and the corrupted $h + δ$ (red) — the red arrow is the unwanted kick.

Fix: set $W_{up} = 0$ at init. Then $δ = 0 \cdot σ (W_{down} h) = 0$ , so $Adapter (h) = h$ exactly — the residual makes the module start as identity, training from the known-good pretrained state. Why it matters? This is the parent's step-3 "init-to-identity" made visible; skipping it disturbs the forward pass and risks the very forgetting PEFT was meant to avoid.

Verify: with $W_{up} = 0$ , $δ = 0 \Rightarrow$ output $= h$ . Any nonzero $W_{up}$ gives $δ \neq = 0$ . The identity property holds iff $W_{up} = 0$ . ✓

Recall Quick self-test over the matrix

Adapter params for $d = 1024, L = 24, k = 2, r = 16$ ? ::: $\approx 1.57$ M ( $0.449%$ of a 350M backbone). What does $r = 0$ produce? ::: The exact identity, zero trainable params. Above which $r$ does an adapter cost more than a full $d \to d$ map? ::: $r > d /2$ . Rank of a $σ$ -less adapter correction with $r = 16$ ? ::: At most 16 (rank collapse). Prefix params for $d = 768, p = 10, L = 12$ ? ::: $184, 320 \approx 0.168%$ . Attention-work ratio going $p = 10 \to 200$ at $n = 50$ ? ::: $\approx 17.4 \times$ (quadratic $(n + p)^{2}$ ). Disk saving serving 100 customers, PEFT vs full FT? ::: $\approx 97 \times$ less. Why init $W_{up} = 0$ ? ::: Makes $δ = 0$ so the adapter starts as exact identity, preserving pretrained behaviour.

Worked examples — Adapter layers and prefix tuning

The scenario matrix

Adapter examples

Prefix examples

Word problem & exam twist