Adapter layers and prefix tuning

WHY does this exist?

WHAT is the trade we make? We give up the ability to freely reshape every weight, and bet that a low-capacity add-on is enough to specialise a model that already learned general features. Empirically, this bet usually wins.

Adapter Layers

HOW it is derived (from first principles)

We want a module that (1) can be inserted without breaking the pretrained forward pass, (2) has few parameters, (3) can express a useful transformation.

Cheap capacity → factorize. A full learned map $\mathbb{R}^d\to\mathbb{R}^d$ $R^{d} \to R^{d}$ costs $d^2$ $d^{2}$ params. Force it through a bottleneck $r$ $r$ : down-project $d\to r$ $d \to r$ , then up-project $r\to d$ $r \to d$ . Params drop to $2dr$ $2 d r$ . With $r\ll d$ $r ≪ d$ , that's tiny.
- Why this step? $2dr \ll d^2$ when $r\ll d$ . For $d=1024, r=16$ : $2dr=32{,}768$ vs $d^2\approx 10^6$ .
Add nonlinearity. Two stacked linears collapse to one linear map. Insert $\sigma$ $σ$ so the adapter can learn a nonlinear correction.
- Why? Without $\sigma$ , $W_{\text{up}}W_{\text{down}}$ is just a rank- $r$ linear map — less expressive.
Wrap in a residual. Initialize $W_{\text{up}}\approx 0$ $W_{up} \approx 0$ so at the start $\text{Adapter}(h)\approx h$ $Adapter (h) \approx h$ .
- Why? If the adapter starts as near-identity, the frozen model's behaviour is preserved on step 0, so training starts from a known-good point and won't destroy pretrained knowledge.

Prefix Tuning

HOW / WHY it works

The real token queries $Q$ now attend over extra learned key/value slots. Since softmax mixes over all keys, the prefix can inject task-specific information into every position.
Only $P_k, P_v$ (size $2 p d$ per layer) are trained; $W_Q, W_K, W_V$ stay frozen.
Why parametrize $K,V$ and not just prepend embeddings? Prepending raw embeddings ("prompt tuning") only affects layer 0; directly writing keys/values at every layer gives more control and trains more stably. (In practice a small MLP reparametrizes the prefix during training for stability.)

Adapters vs Prefix Tuning (80/20 core)

	Adapters	Prefix tuning
Where params go	inside FFN blocks	attention K/V
Trainable form	$W_{\text{down}}, W_{\text{up}}$	$P_k, P_v$
Inference cost	extra tiny FFN per layer	longer effective sequence
Init trick	$W_{\text{up}}\to 0$ (identity)	reparam MLP for stability
Params	$\approx 2Lk\,dr$	$\approx 2Lpd$

The 20% you must know: both freeze the backbone; adapters add a bottleneck FFN with residual init-to-identity; prefix tuning prepends learned key/value vectors; both train ~0.1–3% of weights and give one shared model + tiny per-task deltas.

Common mistake Steel-manning the wrong ideas

"Prefix tuning just adds real prompt words." Why it feels right: prompting with words does help. Fix: prefix tuning learns continuous vectors in K/V space at every layer, not discrete tokens — far more expressive and not limited to the vocabulary.

"Adapters replace the FFN sublayer." Why it feels right: both are FFN-shaped. Fix: adapters are added after a sublayer with a residual; the original sublayer still runs. Nothing is replaced.

"More trainable params always beats fewer, so full fine-tuning is safer." Why it feels right: more capacity = lower training loss. Fix: on small datasets full fine-tuning overfits and forgets; PEFT's low capacity is a regularizer and often matches or beats full FT while being 100× cheaper to store.

"Set adapter $W_{\text{up}}$ randomly like any layer." Why: that's how we init nets. Fix: random init makes the adapter a noise injection at step 0, disturbing the pretrained forward pass. Init near-zero so it starts as identity.

Recall Feynman: explain to a 12-year-old

Imagine a super-smart robot that already read every book. You don't want to re-teach it everything just to do your homework. So you clip a tiny helper gadget onto it (adapter) or hand it a secret cheat-note it reads before every answer (prefix). The robot's big brain stays exactly the same — you only train the tiny gadget or the little note. That's cheap, fast, and you can keep 100 different gadgets for 100 different jobs, all clipping onto the same one robot.

Flashcards

What problem does PEFT (adapters / prefix tuning) solve?

Full fine-tuning needs a full model copy per task (O(W) params); PEFT freezes the backbone and trains only ~0.1–3% new params, one shared model + tiny deltas.

Write the adapter forward equation.

\text{Adapter}(h)=h+W_{\text{up}}\,\sigma(W_{\text{down}}h)

with

W_{\text{down}}\in\mathbb{R}^{r\times d}

W_{\text{up}}\in\mathbb{R}^{d\times r}

Why does an adapter use a bottleneck dimension

r\ll d

To cut params from

d^2

(full map) to

2dr

; low capacity is cheap and acts as a regularizer.

Why does an adapter need a nonlinearity

\sigma

Two stacked linear maps collapse into one;

\sigma

lets the adapter learn a nonlinear correction (otherwise only a rank-

r

linear map).

Why initialize

W_{\text{up}}\approx 0

in an adapter?

\text{Adapter}(h)\approx h

at step 0 (near-identity), preserving pretrained behaviour and starting from a known-good point.

Approx trainable params for adapters across L layers?

\approx 2Lk\,dr

(k adapters/layer), independent of vocab and sequence length.

In prefix tuning, what exactly is learned?

Per-layer key/value prefix matrices

P_k,P_v\in\mathbb{R}^{p\times d}

; queries stay real, K and V get the learned prefix prepended.

Prefix-tuning attention equation?

K'=[P_k;K],\ V'=[P_v;V]

, then

\mathrm{softmax}(QK'^\top/\sqrt d)V'

Prefix tuning vs prompt tuning?

Prompt tuning edits only layer-0 input embeddings; prefix tuning injects learned K/V at every layer — more expressive/stable.

Prefix param count formula?

\approx 2Lpd

(keys+values over L layers, prefix length p).

Steel-man: why is full fine-tuning NOT always better than PEFT?

On small data it overfits and forgets; PEFT's low capacity regularizes and often matches FT at ~100× lower storage.

Compute cost of a length-p prefix on a length-n sequence?

Attention becomes

O((n+p)^2)

— longer effective sequence per token.

Connections

LoRA and low-rank adaptation — another PEFT: trains low-rank weight deltas instead of inserted modules.
Prompt tuning and soft prompts — the layer-0-only special case of prefix tuning.
Transformer attention mechanism — prefix tuning edits its K/V.
Residual connections — the trick behind adapter init-to-identity.
Catastrophic forgetting — why freezing the backbone helps.
Full fine-tuning vs PEFT — the storage/compute trade-off.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek bada LLM already language ko samajh chuka hai — usne internet ka bahut saara text padh liya. Ab agar tum har naye task ke liye poore model ko dobara train karo (full fine-tuning), to har task ke liye ek alag 7B-parameter copy banani padegi. Ye storage aur compute dono mein bahut mehenga hai. Iska smart solution hai PEFT: model ko freeze kar do, aur sirf thode se naye chhote parameters train karo jo model ko task ki taraf steer karein.

Do popular tarike hain. Adapter ek chhota bottleneck FFN hai jo har layer ke andar daala jaata hai: pehle dimension ko $d$ se $r$ (jahan $r$ bahut chhota hai) tak neeche laate hain, phir nonlinearity, phir wapas $d$ tak up-project, aur ek residual +h add karte hain. $W_{up}$ ko lagbhag zero se init karte hain taaki shuru mein adapter almost identity ho — matlab pretrained model ka behaviour kharab na ho. Params sirf $2dr$ — poore model ka ~1%.

Prefix tuning mein hum weights ko haath nahi lagate. Har attention layer mein hum kuch learnable "nakli tokens" ke key/value vectors ( $P_k, P_v$ ) prepend kar dete hain. Real tokens ke queries in extra learned keys/values pe bhi attend karte hain, jisse har position pe task-specific information inject ho jaati hai. Ye discrete prompt words nahi hain — ye continuous vectors hain, har layer mein, isliye zyada powerful.

Core baat yaad rakho: "Freeze the giant, train the tiny." Ek hi bada model rakho, aur har task ke liye sirf ek chhota MB-size delta save karo. Isse 100 customers ko serve karna sasta ho jaata hai, aur choti dataset pe overfitting/forgetting bhi kam hoti hai kyunki capacity kam hai (natural regularizer).

WHY does this exist?

Adapter Layers

HOW it is derived (from first principles)

Prefix Tuning

HOW / WHY it works

Adapters vs Prefix Tuning (80/20 core)

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Pretraining & Fine-Tuning LLMs

Connections