LoRA and QLoRA: Visual walkthrough — AI-ML deep dive

A visual walkthrough of why a giant weight update can be squeezed into two skinny matrices — built from absolute zero, one picture per step.

This is the picture-first companion to LoRA and QLoRA. We assume you have seen a grid of numbers (a matrix) and nothing else. Every symbol is earned before it is used.

If you want the neighbouring ideas: Matrix Rank and Low-Rank Approximation, Full Fine-Tuning vs PEFT, Adam Optimizer memory cost, Quantization of Neural Networks, Attention Q K V Projections.

Step 1 — What a linear layer actually is

Call the input list $x$ . It has $k$ numbers, so we say $x \in R^{k}$ — read that as " $x$ is a point in $k$ -dimensional space", i.e. a list of $k$ real numbers.

The recipe grid is $W_{0}$ . It has $d$ rows and $k$ columns, written $W_{0} \in R^{d \times k}$ . The output is:

$h = W_{0} x$

$x$ ::: the incoming list of $k$ numbers.
$W_{0}$ ::: the frozen, already-trained recipe grid ( $d$ rows $\times$ $k$ columns).
$h$ ::: the outgoing list of $d$ numbers (one per row of $W_{0}$ ).

WHY this matters: every number inside $W_{0}$ is a "weight" we could in principle change during fine-tuning. There are $d \times k$ of them — for a real layer that is millions.

PICTURE: the grid on the left, the input column on its right, the output column below — each output cell is one row of the grid combined with $x$ .

Step 2 — Fine-tuning wants to add a correction grid

$W = W_{0} + Δ W$

$W_{0}$ ::: the original grid, kept exactly.
$Δ W$ ::: the correction grid — same shape $d \times k$ — this is what fine-tuning learns.
$W$ ::: the new, adapted grid.

The Greek letter $Δ$ ("delta") is the standard maths symbol for "the change in". So $Δ W$ literally reads "the change in $W$ ".

WHY this is a problem: $Δ W$ is also $d \times k$ . Learning it fully costs as much memory as $W_{0}$ itself — and training memory (via Adam) is several times worse. We have bought nothing.

PICTURE: two same-size grids stacked, a "+" between them, giving a same-size result. Notice how fat $Δ W$ is — that fatness is the enemy.

Step 3 — The rank insight: many grids are secretly thin

Look at the picture: a $4 \times 4$ grid where every column is a multiple of $(1, 2, 3, 4)$ . Sixteen numbers, but you could rebuild all of them from one column $(1, 2, 3, 4)$ and four multipliers. That grid has rank 1.

WHY we care: experiments show the correction $Δ W$ during fine-tuning behaves like a low-rank grid — it only needs a handful of independent directions, not $min (d, k)$ of them. The task-specific change is genuinely thin. (Deeper background: Matrix Rank and Low-Rank Approximation.)

PICTURE: on the left a full "rank-many" grid of scattered colours; on the right a rank-1 grid whose columns are obviously the same pattern rescaled.

Step 4 — A thin grid factors into two skinny grids

$Δ W \approx B A, B \in R^{d \times r}, A \in R^{r \times k}$

$r$ ::: the rank we allow — a small number we choose, like $8$ . The symbol $≪$ means "much smaller than", so $r ≪ min (d, k)$ .
$B$ ::: tall and skinny — $d$ rows, only $r$ columns.
$A$ ::: short and wide — only $r$ rows, $k$ columns.
$B A$ ::: multiply them back and you get a full $d \times k$ grid whose rank is at most $r$ .

Why this factorization and not something fancier? Because a product $B A$ can produce any rank- $\leq r$ grid and nothing wider. It is the smallest honest description of a rank- $r$ correction — exactly the restriction we wanted.

WHY the win is huge: we now train $r (d + k)$ numbers instead of $d \cdot k$ .

$LoRA r (d + k) ≪ full d \cdot k$

For $d = k = 4096, r = 8$ : full $= 16, 777, 216$ , LoRA $= 65, 536$ — a 256× shrink.

PICTURE: the fat correction grid on the left; an arrow to a tall skinny $B$ times a short wide $A$ on the right, with the tiny shaded areas showing how little we now store.

Step 5 — Plugging the factors into the forward pass

Read it left to right:

$W_{0} x$ ::: the original frozen layer, untouched.
$B A x$ ::: the low-rank correction applied to the same input. Compute it right-to-left: first $A x$ (a tiny $r$ -length list), then $B$ turns it back into a $d$ -length list. Cheap.
$α$ ::: a scaling knob (a plain number, e.g. $16$ ).
$\frac{α}{r}$ ::: divides by the rank so the update's strength doesn't balloon when you pick a bigger $r$ .

WHY divide by $r$ : as $r$ grows, $B A x$ sums more terms and naturally gets larger. Dividing by $r$ keeps the effective push roughly constant, so you don't re-tune the learning rate every time you change rank.

PICTURE: input $x$ flows into two parallel paths — the wide frozen path $W_{0}$ and the narrow bottleneck path $A \to B$ — and their outputs add.

Step 6 — The zero-init edge case (why $B$ starts empty)

At step 0: $Δ W = B A = 0 A = 0 \Rightarrow h = W_{0} x$

$0$ ::: the all-zeros grid; zero times anything is zero.

WHY this exact choice: if both $A$ and $B$ were random, the very first output would be $W_{0} x + (random garbage)$ — we'd inject noise into a carefully trained model and see a sudden loss spike. Starting from $Δ W = 0$ means the model at step 0 is identical to the pretrained one, and the first gradient step is a pure improvement direction.

PICTURE: two loss-vs-step curves — the zero-init curve starts at the pretrained loss and descends smoothly; the both-random curve leaps up first, then struggles back down.

Step 7 — The merge: paying zero at inference

$W \leftarrow W_{0} + \frac{α}{r} B A$

$\leftarrow$ ::: "becomes" — overwrite $W_{0}$ with the merged grid.
After merging, $W$ is one plain $d \times k$ grid again. No bottleneck path, no extra work at inference.

WHY this beats adapter layers: adapters bolt on extra sub-networks that always run, adding latency. LoRA's correction can be absorbed into the weight it corrects, so a served model runs exactly as fast as the original. (The one exception: if you swap many different adapters on the fly, you keep them separate and skip merging.)

PICTURE: the two-path diagram collapses via an arrow into a single merged grid identical in shape to the original.

The one-picture summary

Everything above in a single flow: the fat forbidden $Δ W$ , its factoring into skinny $B$ and $A$ , the parallel forward pass, and the final merge back to one grid.

Recall Feynman: the whole walkthrough in plain words

A layer is a grid of numbers that mixes your inputs. To adapt it we want to add a correction grid, but a full correction is just as huge as the original — no savings. The rescue: the correction you actually need is "thin" (low rank), meaning it hides only a few real directions. A thin grid always splits into a tall-skinny grid $B$ times a short-wide grid $A$ , and those two together are tiny. We run the frozen original and the little $A \to B$ bottleneck side by side and add their outputs. We start $B$ at all zeros so the correction is invisible on step one and we never shock the trained model. When training finishes we fold the correction back into the original grid, so serving the model costs nothing extra. That is LoRA: two small decorations you can bolt on, learn cheaply, and melt back in.

Recall

Rank of a grid whose every column is a multiple of one column? ::: 1. Shapes of $B$ and $A$ in $Δ W \approx B A$ ? ::: $B$ is $d \times r$ , $A$ is $r \times k$ . Parameters trained per matrix vs full? ::: $r (d + k)$ instead of $d \cdot k$ . Why is $B$ initialised to zero? ::: So $B A = 0$ at step 0 and the model starts identical to pretrained. Why divide by $r$ ? ::: Keeps update strength stable as rank changes, so the learning rate need not be retuned. Inference cost after merging $W_{0} + \frac{α}{r} B A$ ? ::: Zero extra — it's one ordinary grid again.