Foundations — Parameter-efficient fine-tuning (PEFT)

This page builds every symbol used in the parent PEFT note from absolute zero. If a line in that note used a letter you didn't recognise, it is defined here first, with a picture.

1. A vector $x$ — a list of numbers with a direction

$x=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_k\end{array}\right]$

• $x_1,x_2,\dots$ are the entries (the individual numbers).
• $k$ is how many entries there are — the dimension.
• $\mathbb{R}$ (a fancy "R") just means "the real numbers" — any decimal you like, positive or negative.

The picture. A vector of 2 numbers is an arrow on a flat page: the first number is how far right, the second how far up. A vector of $k$ numbers is the same idea in $k$ directions at once (we can't draw more than 3, but the idea is identical).

Why the topic needs it. In an LLM, each token (word-piece) is turned into a vector $x$ — a list of, say, 4096 numbers describing its meaning. Everything the model does is push these vectors around.

2. A matrix $W$ — a machine that transforms vectors

$W=\left[\begin{array}{cccc}{W}_{11}& W_{12}& \cdots & W_{1k}\\ W_{21}& W_{22}& \cdots & W_{2k}\\ ⋮& & \ddots & ⋮\\ W_{d1}& \cdots & & W_{dk}\end{array}\right]$

• The size is written rows $\times$ columns: $d\times k$.
• $W_{ij}$ = the number in row $i$, column $j$.

The picture. Think of $W$ as a grid of dials. Each dial contributes a little push to the output. A $d\times k$ grid has exactly $d\cdot k$ dials — and that number is why full fine-tuning is expensive.

Why the topic needs it. The parent note's very first line, $h=Wx$, is "the layer's weight matrix $W$ transforms the input vector $x$." Every trainable part of the model is a matrix like this.

3. Matrix–vector multiply $h=Wx$ — how the machine runs

$h_i=W_{i1}{x}_1+W_{i2}{x}_2+\cdots +W_{ik}{x}_k$

• $x$ has $k$ entries → matches the $k$ columns of $W$. (Sizes must line up.)
• $h$ has $d$ entries → one per row of $W$.

Why the topic needs it. The whole LoRA equation $h=Wx+\frac{\alpha }{r}BAx$ is three of these weighted votes added together. If you can read $Wx$, you can read the whole thing.

4. $\Delta W$ — the change we want to make

$W_{\text{new}}=W+\Delta W$

The picture. Lay $\Delta W$ exactly over $W$; add the two grids cell by cell. If a cell of $\Delta W$ is zero, that dial doesn't move.

Why the topic needs it. Fine-tuning = finding a good $\Delta W$. PEFT's entire trick is: don't store $\Delta W$ as a full dense grid — store it cheaply. That "cheaply" is the next idea.

5. Rank — how much independent information a matrix holds

This is the single most important prerequisite, so we go slowly.

Tiny example. Look at these two $3\times 3$ grids:

• Left: row 2 is $2\times$ row 1; row 3 is $3\times$ row 1. There is really only one direction here → rank $1$. It looks like a $3\times 3$ grid of 9 numbers, but it is secretly simple.
• Right: the three rows point in three totally separate directions → rank $3$ (full).

Why the topic needs it. The parent's key claim — "fine-tuning updates lie on a low intrinsic dimension" — means exactly: $\Delta W$ is low rank. It looks big but is secretly simple, so we can store it in far fewer numbers.

6. Factorization $\Delta W=BA$ — storing a low-rank matrix cheaply

Why two matrices reproduce a $d\times k$ grid. The rule "rows $\times$ columns must match" lets $B$ ($d\times r$) multiply $A$ ($r\times k$): the shared inner size $r$ cancels, leaving $d\times k$. The small waist $r$ is the bottleneck — information is forced through only $r$ channels, which is why the result is guaranteed low rank.

Counting the dials.

For $d=k=4096$, $r=8$:

$d\cdot k=4096\times 4096=16,777,216$ $r(d+k)=8(4096+4096)=65,536$

That is $16,777,216/65,536=256$ times fewer numbers — the parent note's "256×".

Why the topic needs it. This is LoRA's storage trick. See Low-Rank Matrix Factorization for the deeper theory.

7. The remaining Greek and helpers

8. Where each symbol lives in the master equation

Now every piece of the parent's headline formula is defined:

Read left to right: the original weighted vote $Wx$, plus a scaled, low-rank correction $\frac{\alpha }{r}BAx$. Nothing here is unexplained anymore. LoRA typically inserts this on the $q$ and $v$ matrices inside Transformer Attention; the alternative it competes with is Full Fine-Tuning, and its cousin is Prompt Tuning.

Prerequisite map

Equipment checklist