Adapter layers and prefix tuning: Question bank

Definition Every symbol used on this page (self-contained)

So you never have to page-hop:

$d$ = the hidden dimension of the transformer — the length of the vector that represents each token as it flows through a layer (e.g. $768$ in BERT-base).
$r$ = the bottleneck rank of an adapter — a much smaller width the vector is squeezed down to inside the adapter, with $r ≪ d$ (e.g. $16$ or $64$ ).
$L$ = the number of transformer layers stacked in the model (e.g. $12$ ).
$k$ = the number of adapters inserted per layer — usually $1$ or $2$ (one after attention, one after the FFN sublayer).
$n$ = the original sequence length — how many real tokens are in the input.
$p$ = the prefix length — how many learned "fake" key/value slots prefix tuning prepends.
$W_{down} \in R^{r \times d}$ squeezes $d \to r$ ; $W_{up} \in R^{d \times r}$ expands $r \to d$ ; $σ$ is a nonlinearity; $P_{k}, P_{v} \in R^{p \times d}$ are the learned prefix keys/values.

True or false — justify

Both adapters and prefix tuning change the frozen backbone weights during training.

False — both keep every pretrained weight frozen; they only train new add-on parameters (

W_{down}, W_{up}

P_{k}, P_{v}

An adapter with no nonlinearity

σ

can still be as expressive as one with

σ

False — two stacked linears

W_{up} W_{down}

collapse to a single rank-

r

linear map; without

σ

you lose all nonlinear correction power.

Prefix tuning increases the number of real tokens the model can output.

False — the prefix lives only in key/value space; it steers attention but never appears in the output vocabulary or the generated text.

Adapters add trainable parameters that scale with the vocabulary size.

False — adapter cost

\approx 2 L k d r

(

k

= adapters per layer,

L

= layers) depends only on

d

r

L

k

; it is independent of vocab and sequence length.

Because PEFT trains fewer parameters, its training loss is necessarily higher than full fine-tuning's.

False as stated — with fewer trainable degrees of freedom the lower bound on achievable training loss is usually higher, but on a small dataset even a low-capacity model can drive training loss to near zero, so "necessarily higher" is wrong; capacity limits potential, it does not guarantee a gap.

Prefix tuning's compute cost per token is unaffected by the prefix length

p

False — attention becomes

O ((n + p)^{2})

where

n

is the original sequence length, so every extra prefix token lengthens the effective sequence and adds compute for all real tokens.

Initializing adapter

W_{up}

to zero makes the whole adapter useless forever.

False — only step 0 acts as identity; gradients still flow into

W_{down}

(and later

W_{up}

), so the module trains away from identity — see the gradient-flow item below.

An adapter's residual +h is optional decoration.

False — the skip connection is what lets the module start near-identity and preserves the pretrained forward pass; remove it and step 0 injects a near-random shift.

Spot the error

"Prefix tuning just prepends real prompt words to the input, like handcrafted prompting."

The error: it uses discrete tokens. Prefix tuning learns continuous

P_{k}, P_{v}

vectors injected at every layer's attention — far more expressive and not limited to the vocabulary. (See the injection figure below.)

"Adapters replace the FFN sublayer of the transformer with a smaller one."

The error: nothing is replaced. The original sublayer still runs; the adapter is added after it with a residual, correcting the output rather than substituting for it.

"To save memory, set the adapter's up-projection to random init like any fresh layer."

The error: random init means the adapter injects noise at step 0, disturbing the pretrained forward pass. Init

W_{up} \approx 0

so it starts as identity.

"Since prefix tuning only edits attention, we should also feed the prefix into the queries

Q

The error: queries come only from real tokens. The prefix supplies extra keys and values for real queries to attend over; giving the prefix queries would make it attend to itself pointlessly.

"Adapter down-projection is

R^{d \times r}

and up-projection is

R^{r \times d}

The error: the shapes are swapped. Down-project

d \to r

needs

W_{down} \in R^{r \times d}

; up-project

r \to d

needs

W_{up} \in R^{d \times r}

"Prompt tuning and prefix tuning are the same thing."

The error: prompt tuning adds soft embeddings only at layer 0; prefix tuning writes learned key/value vectors at every layer, giving deeper and more stable control.

Why questions

Why factorize the adapter through a bottleneck

r ≪ d

instead of a full

d \times d

map?

A full map costs

d^{2}

params; the bottleneck costs only

2 d r

, which is tiny when

r ≪ d

, and the low capacity also acts as a regularizer (see the regularizer item).

Why does a low-capacity bottleneck actually act as a regularizer (not just "fewer params")?

Squeezing

d \to r \to d

forces every learned correction to live in an

r

-dimensional subspace. The adapter literally cannot represent arbitrary per-example fixes — it can only apply a rank-limited transform shared across all inputs. That constraint is exactly a hypothesis-space restriction (like weight-tying or a low-rank prior), so it prunes the high-frequency, memorize-the-noise solutions that overfitting relies on.

Why does prefix tuning write into keys/values at every layer instead of only prepending embeddings at the input?

Input-only prepending (prompt tuning) affects layer 0 and must propagate upward; writing

K / V

directly at each layer gives per-layer control and trains more stably.

Why is a low-capacity add-on a feature rather than a limitation on small datasets?

Limited capacity resists memorizing noise, so PEFT overfits less and helps avoid Catastrophic forgetting — it regularizes toward the pretrained solution.

Why can the prefix influence every real token even though it sits only at the front?

Because softmax attention mixes over all keys, every real query attends to the prefix key/value slots, so task information leaks into every position.

Why prefer PEFT over keeping 100 full fine-tuned copies for 100 tasks?

Full copies cost

O (W)

storage each (e.g. ~28 GB per 7B model); PEFT keeps one frozen backbone plus a few MB of deltas per task — a shared giant, tiny swappable gadgets.

Why does full fine-tuning risk "forgetting" while PEFT usually does not?

Full FT can overwrite the pretrained weights that store general knowledge; PEFT freezes those weights entirely, so general competence is structurally protected. See Catastrophic forgetting.

Why does putting a small MLP on top of the raw prefix vectors stabilize optimization?

Optimizing

P_{k}, P_{v}

directly means each of the

2 p d

numbers is a free, uncoupled parameter — the loss surface in that raw space is jagged and the good directions are ill-conditioned, so gradients bounce. Feeding a small trainable seed through a shared MLP couples all prefix slots through the same smooth function: updates move a few MLP weights that jointly reshape all slots consistently, so the effective landscape is smoother and better-conditioned. At inference the MLP output is frozen and cached, so it costs nothing.

Edge cases

Walk through why gradients still flow when

W_{up}

starts at zero.

The adapter is

h + W_{up} σ (W_{down} h)

. The gradient of the loss w.r.t.

W_{down}

\frac{\partial L}{\partial out} \cdot W_{up}^{⊤} \cdot σ^{'} (\cdot) \cdot h^{⊤}

— this contains

W_{up}

, so at step 0 it is zero and

W_{down}

doesn't move yet. But the gradient w.r.t.

W_{up}

\frac{\partial L}{\partial out} \cdot σ (W_{down} h)^{⊤}

, which does not contain

W_{up}

and is generally nonzero. So

W_{up}

moves off zero on the first step; once it is nonzero,

W_{down}

starts receiving gradient too. The zero-init only stalls one factor for one step, never permanently.

Show the residual identity at step 0 explicitly.

With

W_{up} = 0

Adapter (h) = h + 0 \cdot σ (W_{down} h) = h + 0 = h

. The output equals the input exactly, so the pretrained forward pass is untouched and training starts from a known-good point.

What if you set the bottleneck to

r = d

(no bottleneck)?

You lose the parameter savings (

2 d^{2}

params, comparable to a full map) and the rank-limiting regularizer; the adapter is no longer "cheap capacity."

What if prefix length

p = 0

There is no prefix at all —

K^{'} = K

V^{'} = V

— so attention reduces to the ordinary frozen model with zero trainable prefix parameters.

What if you push

p

very large for a simple classification task?

Params and compute grow linearly in

p

(and attention grows as

O ((n + p)^{2})

n

= real tokens) for little gain; a simple task saturates capacity quickly, so extra prefix length mostly wastes compute.

What if you accidentally leave the backbone unfrozen while adding adapters?

You are effectively doing full fine-tuning plus extra modules — losing the storage savings and reintroducing the forgetting/overfitting risks PEFT was meant to avoid.

What distinguishes adapters from LoRA since both use a low-rank

r

An adapter is a separate nonlinear bottleneck block inserted with its own residual; LoRA adds a low-rank linear delta directly onto existing weight matrices with no extra nonlinearity or inserted module.

What is the effect on inference latency of adapters versus prefix tuning?

Adapters add a small extra FFN per layer (constant per token); prefix tuning lengthens the effective sequence, so its overhead grows with attention's

O ((n + p)^{2})

cost.

Visualize it

The figure below shows where the prefix slots enter attention: real queries $Q$ (from $n$ real tokens) attend over an augmented key/value set — the $p$ learned prefix slots concatenated in front of the $n$ real keys/values. Look at how every real query row reaches leftward into the plum prefix block.

Figure — Adapter layers and prefix tuning

This second figure traces the adapter's zero-init residual: the vector $h$ takes the identity skip (teal) while the bottleneck branch (orange) multiplies by $W_{up} = 0$ and contributes nothing at step 0, so the sum is exactly $h$ .

Recall One-line self-test

Cover everything above and answer: "Both PEFT methods freeze the backbone; adapters add a nonlinear bottleneck FFN with residual init-to-identity; prefix tuning prepends $p$ learned key/value vectors at every layer; both train ~0.1–3% of weights." If you can rebuild that sentence with reasons, you own this page.