Parameter-efficient fine-tuning (PEFT): Visual walkthrough

Step 0 — What is a "weight matrix" anyway?

WHAT. A neural-network layer takes a list of numbers (a vector) and turns it into another list of numbers. The rule it uses is: multiply the input by a grid of numbers. That grid is the weight matrix $W$ .

WHY start here. Every symbol later ( $W$ , $Δ W$ , $B$ , $A$ ) is a grid of numbers. If you can't picture a grid multiplying a list, nothing after makes sense.

PICTURE. Look at Figure s01. On the left is the input list $x$ — think of it as $k$ numbers stacked in a column (here $k = 4$ ). The big orange grid is $W$ : it has $d$ rows and $k$ columns (here $d = 3$ , $k = 4$ ). Multiplying gives $h$ , a new column of $d$ numbers.

$h = grid, d \times k W list, k x = list, d h$

Figure — Parameter-efficient fine-tuning (PEFT)

Step 1 — Fine-tuning is just nudging the grid

WHAT. To adapt the model to a new task, fine-tuning changes every number in $W$ a little. Call the pile of changes $Δ W$ (read "delta W" — delta means "the change in"). The new grid is $W + Δ W$ .

WHY. Any adaptation, no matter how it's computed, ends up as "the old grid plus some correction grid." So the honest thing to ask is: what does that correction grid look like, and do we really need to store all of it?

PICTURE. Figure s02 shows the old grid $W$ (orange) and the correction grid $Δ W$ (teal) laid on top. Crucially $Δ W$ is the same shape as $W$ — also $d \times k$ . Adding them position-by-position gives the tuned grid.

Step 2 — The key observation: the correction is "flat"

WHAT. We now claim $Δ W$ is low rank. "Rank" measures how many genuinely different directions a grid contains. A high-rank grid is rich and full; a low-rank grid is repetitive — its rows are mostly echoes of a few basic rows.

WHY this tool, and why now. We need a way to say "this grid, though big, carries little real information." The mathematical name for that is low rank. Experiments show the fine-tuning correction $Δ W$ genuinely is low rank — the useful change lives in only a handful of directions. So storing all $d \times k$ numbers is wasteful.

PICTURE. Figure s03 contrasts two $d \times k$ grids of the same size. On the left, a full-rank grid: every row points a different way (imagine many independent arrows). On the right, a rank-2 grid: every row is just a blend of two "master" directions — the plum arrows. Same footprint, far less information.

Step 3 — A low-rank grid = two skinny grids multiplied

WHAT. Here is the algebra fact that makes LoRA cheap: any grid of rank $\leq r$ can be written as a product of two skinny grids, $Δ W = B A .$

WHY. If $Δ W$ only has $r$ building-block directions, we can store just those directions ( $B$ ) and just the recipe for mixing them ( $A$ ) — instead of the whole fat grid. Multiplying $B$ by $A$ rebuilds $Δ W$ on demand.

PICTURE. Figure s04 shows the factorization visually. The tall thin orange grid $B$ is $d \times r$ — it holds $r$ column-directions. The wide thin teal grid $A$ is $r \times k$ — it holds, for each of the $k$ inputs, how much of each direction to use. Their product fills the whole $d \times k$ shape, but was built from far fewer numbers.

Step 4 — Count the numbers you actually store

WHAT. Compare the storage of the fat correction versus the two skinny grids.

WHY. This is the entire payoff of the low-rank move, so we make it concrete.

PICTURE. Figure s05 plots how many numbers you store as the rank $r$ grows, for $d = k = 4096$ . The flat orange line is full $Δ W$ ( $d \cdot k$ numbers — it never changes). The rising teal line is LoRA ( $r (d + k)$ ). Notice how far below the orange line the teal one sits for small $r$ .

Step 5 — The forward pass, with the scaling knob

WHAT. Put it together. The frozen grid does its usual job; the skinny pair adds a correction, scaled by $\frac{α}{r}$ .

WHY the extra $\frac{α}{r}$ ? When you make $r$ bigger, the product $B A$ tends to produce bigger numbers (more terms are summed). Dividing by $r$ cancels that growth, so changing capacity ( $r$ ) doesn't secretly change how loud the correction is. The knob $α$ lets you set the loudness independently.

PICTURE. Figure s06 is the two-path diagram: input $x$ splits, goes through frozen $W$ (orange, locked) on the top path and through $A$ then $B$ (teal, trainable) on the bottom path; the bottom path is scaled by $\frac{α}{r}$ ; the two outputs are added to make $h$ .

Step 6 — The degenerate start: $B = 0$ (why nothing breaks)

WHAT. We initialize $A$ with small random numbers and set $B$ to all zeros. At the very first training step, $B A = 0$ , so the correction path outputs nothing — the model is exactly the pretrained one.

WHY this matters (the degenerate case done right). If we had started with a random non-zero correction, step 0 would already corrupt the pretrained knowledge — a form of forgetting. Starting from $B A = 0$ means training can only add to what's already known, gently, never smash it.

PICTURE. Figure s07 shows the correction magnitude over training: it starts pinned at exactly $0$ (the pretrained model, untouched) and rises smoothly as $B$ learns away from zero. The dotted line marks step 0 where $h = W x$ identically.

Step 7 — Merging: the correction disappears at inference

WHAT. After training, compute $W^{'} = W + \frac{α}{r} B A$ once and store that single grid. Now the layer is just $h = W^{'} x$ — one multiply, no extra path.

WHY. The two-path picture in Step 5 costs a little extra compute per forward pass. But addition of grids can be done ahead of time. Fold the correction into $W$ and the running model is indistinguishable in shape and speed from the original — zero added latency. (Contrast: adapter modules insert extra sequential layers that can't be folded away.)

PICTURE. Figure s08 shows the two-path diagram collapsing into one orange grid $W^{'}$ , with the teal path absorbed. The clock icon shows "extra time = 0."

The one-picture summary

Figure s09 compresses the whole journey onto a single canvas: the frozen grid $W$ , the low-rank correction born from the skinny pair $B, A$ , the scaling $\frac{α}{r}$ , the harmless zero start, and the final merge into $W^{'}$ .

Recall Feynman retelling — the walkthrough in plain words

A layer is just a grid of numbers you multiply your input by (Step 0). To learn a new task, you'd normally nudge every number in that grid — a change we called $Δ W$ (Step 1). But it turns out that change is lazy: it only really moves things along a few directions — it's "low rank" (Step 2). Anything that simple can be rebuilt from two skinny grids, a tall one $B$ and a wide one $A$ , multiplied together (Step 3). Storing those two skinny grids takes hundreds of times fewer numbers than the fat one (Step 4). So we run the frozen original plus a tiny side-path $B A$ , dialed to the right loudness with $α / r$ (Step 5). We start the side-path at exactly zero so the model begins as a perfect copy of the original and can only improve, never forget (Step 6). And when we're done, we can fold the tiny correction back into the big grid so the finished model runs at full speed with nothing extra bolted on (Step 7). Giant frozen brain, tiny cheap sticky notes.

Connections

Parent: PEFT overview — the note this walkthrough zooms into.
Low-Rank Matrix Factorization — Steps 2–3 are exactly this idea.
Full Fine-Tuning — the fat- $Δ W$ baseline we compress.
Quantization — combine with this to get QLoRA.
Transformer Attention — where $B, A$ usually plug in ( $q, v$ projections).
Catastrophic Forgetting — Step 6's zero-init is a guard against it.
Prompt Tuning — the other PEFT branch, not covered here.