Visual walkthrough — Adapter layers and prefix tuning

---

Step 0 — What are we even looking at?

Before symbols, let's fix the picture. A transformer processes a sentence as a stack of layers. Inside each layer live sublayers (attention, then a feed-forward network). Between sublayers travels a bundle of numbers we call the hidden vector $h$.

Look at the blue boxes: those are the pretrained sublayers — we are told to freeze them (never change their weights). The gap between them (the yellow slot) is where we get to insert something new. Our whole job is: design the smallest possible gadget to drop into that yellow slot.

---

Step 1 — The naive gadget, and why it is too big

WHAT. The simplest inserted gadget: a single dense layer that reads $h$ (size $d$) and writes a new vector of size $d$. That is a matrix $M$ of shape $d\times d$.

$\underset{\text{size }d}{\underbrace{h_{\text{new}}}}=\underset{d\times d\text{ matrix}}{\underbrace{M}}\underset{\text{size }d}{\underbrace{h}}$

• $M$ ::: the trainable map — one weight for every (input number, output number) pair
• so $M$ has $d\times d=d^2$ numbers inside it

WHY it fails. For $d=768$ that is $768^2\approx 590,000$ parameters per slot, and there are dozens of slots. We wanted a tiny add-on; $d^2$ is not tiny. We need the same "reads $d$, writes $d$" behaviour with far fewer knobs.

PICTURE. The dense connection is a fully-filled grid — every input wire touches every output wire.

Count the red connection lines: $d\times d$ of them. That density is exactly the cost. The fix in Step 2 is to force the signal through a narrow waist.

---

Step 2 — Squeeze through a bottleneck (the factorization)

WHAT. Instead of going $d\to d$ directly, go $d\to r\to d$ where $r$ is a small number ($r\ll d$, read "$r$ much less than $d$", e.g. $r=64$). Two matrices do this:

$h\stackrel{W_{\text{down}}}{\to }\underset{\text{size }r}{\underbrace{z}}\stackrel{W_{\text{up}}}{\to }{h}_{\text{new}}$

• $W_{\text{down}}\in {\mathbb{R}}^{r\times d}$ ::: squeezes the wide vector ($d$) down to a thin one ($r$). Shape $r\times d$.
• $z$ ::: the squeezed "summary" — only $r$ numbers survive the waist
• $W_{\text{up}}\in {\mathbb{R}}^{d\times r}$ ::: expands the thin vector back up to width $d$. Shape $d\times r$.

WHY this shape. We call $r$ the bottleneck dimension because the signal must pass through this narrow neck. Counting the knobs:

$\underset{W_{\text{down}}}{\underbrace{r\cdot d}}+\underset{W_{\text{up}}}{\underbrace{d\cdot r}}=2dr\ll d^2(\text{when }r\ll d)$

For $d=768,r=64$: $2dr=98,304$ versus $d^2=590,000$ — a 6× shrink here, and it grows to a huge shrink as $d$ climbs. This is the whole reason PEFT is cheap. (Compare LoRA and low-rank adaptation, which factorizes a weight update the same way.)

PICTURE. The dense grid becomes an hourglass: many wires in, few in the middle, many out.

Notice the two thin bundles of lines (each $d\times r$) replace the one fat bundle ($d\times d$). The pinch point in the middle is $z$, the size-$r$ waist.

---

Step 3 — Why two matrices are not enough on their own

WHAT. You might think we're done: down then up. But there is a trap. If we just multiply the two matrices with nothing between them:

$W_{\text{up}}(W_{\text{down}}h)=\underset{\text{one }d\times d\text{ matrix}}{\underbrace{(W_{\text{up}}{W}_{\text{down}})}}h$

• $W_{\text{up}}{W}_{\text{down}}$ ::: multiplying two matrices gives back a single $d\times d$ matrix

WHY this is a problem. The gadget collapses back into one linear map — exactly the plain matrix from Step 1, just secretly rank-limited. All that bottleneck machinery bought us nothing new in behaviour; a straight line is a straight line no matter how you factor it. To bend the map — to let it react differently to different inputs — we must break the chain with something non-straight.

THE TOOL — why a nonlinearity, and why here. We insert $\sigma$, a nonlinearity (a function that is not a straight line, like ReLU: $\sigma (x)=\max (0,x)$). We choose a nonlinearity specifically because it is the one thing that stops two linear maps from fusing. It answers the question: "how do I make the composition genuinely more expressive than a single matrix?"

$\text{gadget}(h)=W_{\text{up}}\sigma (W_{\text{down}}h)$

• $\sigma$ ::: applied to each of the $r$ numbers in $z$; zeros out the negatives (for ReLU), putting a kink in the map

PICTURE. A straight ramp (linear) versus a bent ramp (with ReLU).

The kink is what carries new information. The green bent curve can approximate task-specific corrections the red straight line never could.

---

Step 4 — Don't overwrite the giant: the residual wrapper

WHAT. Right now the gadget replaces $h$ with a brand-new vector. But the frozen model already computed a good $h$! If we throw it away, we've broken the pretrained forward pass. So we add the gadget's output to the original $h$ instead of replacing it:

$\text{Adapter}(h)=\underset{\text{keep the good pretrained vector}}{\underbrace{h}}+\underset{\text{a small learned correction}}{\underbrace{W_{\text{up}}\sigma (W_{\text{down}}h)}}$

• the + h ::: the residual/skip connection (same idea as in a residual block)
• the second term ::: a nudge added on top, not a replacement

WHY. The adapter now only has to learn a small difference, not re-derive the whole thought. And it sets up the crucial init trick in Step 5.

PICTURE. The highway metaphor: $h$ drives straight through on the main road; the adapter is a small side-loop that merges a correction back in.

If the side-loop contributes zero, the value on the highway is completely unchanged. That property is the key to Step 5.

---

Step 5 — Start as the identity (the zero-init trick)

WHAT. At training step 0, before the adapter has learned anything, we want it to do nothing — literally pass $h$ through untouched. We achieve this by initializing $W_{\text{up}}\approx 0$ (all zeros).

$\text{Adapter}(h)=h+\underset{=0}{\underbrace{0\cdot \sigma (W_{\text{down}}h)}}=h$

• $W_{\text{up}}=0$ ::: whatever $\sigma (W_{\text{down}}h)$ is, multiplying by a zero matrix erases it
• so at step 0: $\text{Adapter}(h)=h$ exactly — a perfect identity

WHY. If we instead used random init (the usual way to start a network), the adapter would inject noise into every layer on step 0, scrambling the frozen model's carefully-learned features. Zero-init means training starts from the known-good pretrained model and only departs from it as gradients demand. This directly fights forgetting.

PICTURE. Two training trajectories: random-init (starts far from good, must climb back) versus zero-init (starts at the good model, only improves).

The green path (zero-init) never leaves the safe basin; the red path (random-init) starts in the wilderness. This is why PEFT is stable.

---

Step 6 — Edge cases: what happens at the extremes of $r$?

We must not leave the reader guessing at the boundary values of the bottleneck $r$.

The yellow curve is parameter count $2dr$; it rises linearly with $r$. The dashed line marks the naive $d^2$ budget — cross it (at $r=d/2$) and the adapter is no longer saving anything. The sweet spot is the green band near the left: enough capacity to help, cheap enough to matter.

---

The one-picture summary

Read it left to right: the frozen highway carries $h$; a thin side-loop squeezes $d\to r$ (blue), bends it with $\sigma$ (yellow kink), expands $r\to d$ (blue), and merges the correction back with a + (green). $W_{\text{up}}$ starts at zero (red label) so the loop is silent on step 0.

Recall Feynman: the whole walkthrough in plain words

We have a genius robot we're not allowed to rewire. We want to add a tiny tweak between two of its brain-modules. A full tweak-box would need a wire from every input to every output — way too many wires. So we funnel the signal through a narrow waist: many wires squeeze into a few, then fan back out. But two funnels in a row are secretly just one plain box, so we put a kink in the middle (the nonlinearity) to make it genuinely new. We don't let the tweak-box replace the robot's thought — we add its little correction on top (the +), like scribbling a note in the margin instead of rewriting the page. And on day one we set the output funnel to all zeros, so the box contributes nothing and the robot behaves exactly as before — then it slowly learns to help. Make the waist too wide and you've saved no wires; make it zero and the box does nothing forever. Somewhere in between: a tiny, cheap, stable helper. That is an adapter.

Recall Quick self-check

Why can't we skip $\sigma$? ::: Two stacked linear maps collapse into a single $d\times d$ linear map — no more expressive than the naive Step-1 gadget. Why + h instead of replacing $h$? ::: To preserve the frozen model's forward pass and only learn a small correction (residual). Why init $W_{\text{up}}=0$? ::: So $\text{Adapter}(h)=h$ at step 0 — a safe identity start that avoids injecting noise into the pretrained model. At $r=d$, what goes wrong? ::: Param count $2dr=2d^2$ exceeds even the naive $d^2$ map — the efficiency is lost.

---

Related: Prompt tuning and soft prompts · Transformer attention mechanism · LoRA and low-rank adaptation