Foundations — LoRA and QLoRA

This page assumes nothing. Before you read LoRA and QLoRA proper, we build every symbol it uses — one brick at a time. Each symbol gets: plain words → a picture → why the topic needs it.

1. A vector $x$ — a list of numbers

In an LLM, $x$ is the "state" of a token as it flows through the network — a list of, say, 4096 numbers describing what the model currently "thinks" about that word.

• Picture: a tall thin strip of numbers (see figure below, left).
• Why the topic needs it: every layer of the model takes such a list in and produces a new list out. $x$ is the input to the layer LoRA modifies.

We write $x\in {\mathbb{R}}^k$, read "$x$ lives in the space of $k$ real numbers" — meaning $x$ is a list of $k$ ordinary numbers. $\mathbb{R}$ is the symbol for "all real numbers" (any decimal, positive or negative).

2. A matrix $W$ — a grid that transforms vectors

Look at the figure. A matrix is a machine: feed in a vector of length $k$ (matching its columns), get out a vector of length $d$ (matching its rows). The rule for one output number is: march along a row, multiply each entry by the matching entry of $x$, add them up.

• Why the topic needs it: a linear layer is a matrix. The whole of LoRA is about how to cheaply change this grid $W$.

3. Dimensions $d$, $k$, and why the layer is huge

$d$ and $k$ are just the height and width of the grid.

Multiply $d\times k$ to get the parameter count of one matrix. Keep this multiplication in mind — it is the villain LoRA defeats.

4. The frozen base $W_0$ and the update $\Delta W$

The new weight is simply: $W=W_0+\Delta W$

• Picture: two grids stacked and added cell by cell (right side of figure s01).
• Why the topic needs it: LoRA never touches $W_0$ (it "freezes" it). All the action is in cheaply representing $\Delta W$.

5. Rank — how "thin" a grid really is

This is the concept everything hinges on.

Look at the figure. On the left, a full-rank grid: rows point in many independent directions. On the right, a rank-1 grid: every row is a multiple of the top row — visually it collapses to a single stripe of information smeared across the grid.

• Why the topic needs it: LoRA's founding assumption is that $\Delta W$ has low rank $r$, with $r$ far smaller than $d$ or $k$.

6. The outer product — building a grid from two lists

Before we factor $\Delta W$, we need the tool that makes a rank-1 grid.

Look at the figure: a tall skinny $b$ times a wide skinny $a$ produces a full rectangle. Every grid built this way has rank exactly $1$ (each row is $a$ scaled by that row's $b_i$).

Why the outer product and not, say, ordinary multiplication? We want the smallest building block that produces a full 2-D grid from 1-D lists. The outer product is exactly that atom — and rank $r$ is precisely "how many atoms you needed." This is the reason LoRA can write $\Delta W\approx BA$.

• Parameter payoff: $B$ has $d\times r$ numbers, $A$ has $r\times k$. Together $r(d+k)$ — versus $d\times k$ for the full grid.

7. The skinny matrices $A$, $B$, and the rank $r$

Now the LoRA symbols make sense:

8. The scaling knob $\alpha$

• Picture: a dial turning the injected change up or down.
• Why the topic needs it: without the $\frac{\alpha }{r}$ factor you'd have to re-tune the learning rate every time you changed $r$. The knob decouples them.

9. Quantization — storing numbers in fewer bits (for QLoRA)

• Picture: a smooth number line replaced by 16 tick marks; every value jumps to its closest tick.
• A byte is 8 bits, so 4 bits $=0.5$ byte per number. A 7B-parameter model at 4 bits needs $7\text{B}\times 0.5=3.5$ GB instead of $28$ GB at 32-bit.
• Why the topic needs it: QLoRA stores the frozen $W_0$ in 4-bit to shrink it, and only briefly "dequantizes" (expands it back) during the multiply. See Quantization of Neural Networks.

Because $W_0$ is frozen, this snapping introduces a fixed small error that the trainable $A,B$ simply learn to work around — safe precisely because we never train $W_0$ itself.

10. Optimizer memory — the reason all this exists

Since LoRA only trains the tiny $A,B$, the optimizer only tracks those — a few megabytes. That single fact is the payoff of everything above. See Adam Optimizer memory cost.

Prerequisite map

Equipment checklist

Test yourself — reveal each only after answering aloud.

Ready? Now read LoRA and QLoRA — every symbol on that page is now a picture you already own. For the fine-tuning context see Full Fine-Tuning vs PEFT; for where these matrices sit in the model, Attention Q K V Projections; and for how LoRA differs from Adapter Layers, watch the "merge = zero latency" point.