Foundations — Adapter layers and prefix tuning

This page assumes nothing. If the parent note used a symbol, we build it here from the ground up, in an order where each brick rests on the previous one.

0. What is a "parameter"? (the atom of everything)

Picture a mixing desk with millions of sliders. Each slider is one number. "The model has 7 billion parameters" means 7 billion sliders. Fine-tuning means grabbing sliders and moving them.

• Full fine-tuning = you are allowed to move every slider.
• PEFT (Parameter-Efficient Fine-Tuning) = you glue most sliders in place and add a small handful of new sliders you're allowed to move.

Why the topic needs this: the whole point of adapters and prefixes is how few new numbers you must move. Without the idea of a "parameter", "0.1–3% of parameters" is meaningless.

1. The vector $h$ — "what the model is thinking at one spot"

Read ${\mathbb{R}}^d$ out loud as "a list of $d$ real numbers." The symbol $\in$ means "is a member of / lives in." So $h\in {\mathbb{R}}^d$ = "$h$ is a point in $d$-dimensional space."

Picture: if $d=3$, $h$ is an arrow in 3D space. Real models use $d=768$ or $d=4096$ — we can't draw 768 dimensions, so we always draw 2 or 3 and trust the pattern.

Why the topic needs it: an adapter is a machine that takes $h$ in and gives a corrected $h$ out. The prefix injects extra vectors of the same length $d$. Every formula in the parent is a story about vectors of length $d$.

2. Matrices as machines that reshape vectors

So a matrix is a machine: it eats a length-$n$ vector and spits out a length-$m$ vector. The shape tells you the plumbing:

$\underset{m\times n}{\underbrace{M}}\cdot \underset{\text{length }n}{\underbrace{h}}=\underset{\text{length }m}{\underbrace{\text{out}}}$

Why: every trainable block (adapters, prefix keys/values) is just one or two of these matrices. Counting their entries is how we get "$2dr$ parameters."

3. Why $d^2$ is expensive and $2dr$ is cheap

A full map from length $d$ to length $d$ needs a $d\times d$ matrix = $d^2$ numbers. Squeeze through a bottleneck of size $r$ instead:

• down: $r\times d=dr$ numbers,
• up: $d\times r=dr$ numbers,
• total $=2dr$.

Why the topic needs it: the inequality $r\ll d$ (defined in section 2) is the mathematical reason PEFT is cheap. This inequality is the hinge of the whole idea.

4. Nonlinearity $\sigma$ — the reason two matrices aren't one

Why: this is exactly step 2 of the parent's derivation. See also Full fine-tuning vs PEFT for why low-capacity curved corrections are enough.

5. The residual "+ h" — adding, not replacing

If $f(h)\approx 0$, then $\text{out}\approx h$ and nothing changed — the model behaves exactly as before. This is why we start the adapter with $W_{\text{up}}\approx 0$: at step 0 the correction is zero, so we begin from the known-good pretrained model and gently drift away.

This whole idea is the same one from Residual connections — the adapter is glued into the network the same way a transformer already glues its sublayers.

Why: the "$+h$" in $\text{Adapter}(h)=h+W_{\text{up}}\sigma (W_{\text{down}}h)$ is not decoration — it is why adapters don't destroy pretrained knowledge (see Catastrophic forgetting).

6. Q, K, V — the three roles inside attention

Why the topic needs it: prefix tuning doesn't touch $Q$. It adds extra Keys and Values — the learned matrices $P_k,P_v$ — so every real token's query can also attend to a few trainable "fake" slots. The notation $K^{\prime }=[P_k;K]$ means "stick the prefix keys on the front of the real keys" ( $[\cdot ;\cdot ]$ = concatenation, stacking rows on top of each other). Contrast with Prompt tuning and soft prompts, which only edits the very first layer's input, and LoRA and low-rank adaptation, which edits weights with a bottleneck like the adapter.

How it all feeds the topic

Equipment checklist

Test yourself — cover the right side and answer out loud.