Parameter-efficient fine-tuning (PEFT): Worked examples

Recall the single formula everything on this page uses:

Every symbol above is used below. Nothing new is introduced without a picture.

The scenario matrix

Every LoRA question is really "count parameters" or "track a number through the equation" — but the setup varies along a few axes. Here is the full grid. Each cell gets its own worked example.

Cell	Scenario class	What makes it tricky	Example
C1	Square matrix, standard $r$	The baseline; get the arithmetic reflex	Ex 1
C2	Non-square matrix ( $d \neq = k$ )	$B$ and $A$ have different sizes	Ex 2
C3	Degenerate $r = 0$	LoRA vanishes — sanity limit	Ex 3
C4	Limiting $r = min (d, k)$	LoRA stops saving anything	Ex 4
C5	$α / r$ scaling across ranks	Keep update magnitude fixed	Ex 5
C6	Init step-0 behaviour	$B A = 0$ , forward equals base	Ex 6
C7	Merge & zero-latency	$W^{'} = W + \frac{α}{r} B A$ is one matrix	Ex 7
C8	Storage / multi-task word problem	Real GB-vs-MB budgeting	Ex 8
C9	QLoRA memory exam twist	4-bit base + 16-bit adapters	Ex 9
C10	Break-even rank exam twist	When does LoRA stop being cheaper?	Ex 10

Figure — Parameter-efficient fine-tuning (PEFT)

The figure above shows the sizes: the big frozen square $W$ , and beside it the two skinny strips $B$ (tall, $r$ columns) and $A$ (wide, $r$ rows). All examples are just measuring these shapes or pushing $x$ through them.

C1 — Square matrix, standard rank

Worked example Example 1 — the baseline count

A linear layer has $W \in R^{4096 \times 4096}$ . Apply LoRA with $r = 8$ . How many parameters do you train, and what fraction of the full matrix is that?

Forecast: guess the number of trained params before reading — is it thousands, tens of thousands, or millions?

Shape $B$ and $A$ . $B$ is $d \times r = 4096 \times 8$ ; $A$ is $r \times k = 8 \times 4096$ . Why this step? The formula fixes $B$ 's columns and $A$ 's rows to $r$ ; the other dimension matches $W$ .
Count entries. $B$ has $4096 \times 8 = 32, 768$ ; $A$ has $8 \times 4096 = 32, 768$ . Total $= 65, 536$ . Why this step? Trainable params are exactly the entries of $B$ plus $A$ — nothing in $W$ moves.
Full matrix size. $Δ W$ dense would be $4096 \times 4096 = 16, 777, 216$ . Why this step? This is the thing LoRA replaces; the ratio is the point.
Fraction. $65, 536/16, 777, 216 = 0.00390625 = 0.39%$ .

Verify: general formula $r (d + k) = 8 (4096 + 4096) = 8 \times 8192 = 65, 536$ . ✓ Matches step 2. And $256 \times 65, 536 = 16, 777, 216$ , so LoRA is exactly $256 \times$ smaller.

C2 — Non-square matrix

Worked example Example 2 — when

$d \neq = k$ A projection has $W \in R^{1024 \times 4096}$ (maps a 4096-vector down to 1024). Apply LoRA with $r = 16$ . Count trainable params, and check the two thin matrices multiply back to the right shape.

Forecast: will $B$ and $A$ be the same size this time? (No — watch.)

Shape $B$ . $B$ is $d \times r = 1024 \times 16 = 16, 384$ entries. Why this step? $B$ 's rows always match the output dimension $d = 1024$ .
Shape $A$ . $A$ is $r \times k = 16 \times 4096 = 65, 536$ entries. Why this step? $A$ 's columns always match the input dimension $k = 4096$ . Different from $B$ now!
Total. $16, 384 + 65, 536 = 81, 920$ .
Shape check of $B A$ . $(1024 \times 16) (16 \times 4096) = 1024 \times 4096$ — same shape as $W$ . ✓ Why this step? The inner dimension $r = 16$ cancels; the update must match $W$ to be addable.

Verify: formula $r (d + k) = 16 (1024 + 4096) = 16 \times 5120 = 81, 920$ . ✓ Full $Δ W$ would be $1024 \times 4096 = 4, 194, 304$ , so LoRA is $\approx 51 \times$ smaller here.

C3 — Degenerate input: $r = 0$

Worked example Example 3 — the rank-zero sanity check

What does LoRA do if you set $r = 0$ ? Show the update vanishes and explain why the model is then exactly the base.

Forecast: does the model break, or does it quietly become the frozen model?

Shapes collapse. $B$ is $d \times 0$ and $A$ is $0 \times k$ — matrices with zero columns / rows. Why this step? $r$ literally is the shared dimension; setting it to 0 removes it.
The product. $B A$ is the $d \times k$ zero matrix (an empty sum of outer products $= 0$ ). Why this step? $B A = \sum_{i = 1}^{r} B_{:, i} A_{i, :}$ ; with $r = 0$ there are no terms, so the sum is $0$ .
Forward pass. $h = W x + \frac{α}{0} \cdot 0 \cdot x$ . The $\frac{α}{0}$ is only a red flag on paper — since $B A = 0$ before scaling, the correction is $0$ and $h = W x$ . Why this step? You never actually divide by zero in code because you never build a rank-0 adapter; this is a limiting sanity check showing "no LoRA = base model."

Verify: trainable params $= r (d + k) = 0 \cdot (d + k) = 0$ . Zero rank ⇒ zero new params ⇒ pure base model. ✓

C4 — Limiting case: $r = min (d, k)$

Worked example Example 4 — when LoRA saves nothing

For $W \in R^{4096 \times 4096}$ , push $r$ up to $2048$ . Compare LoRA's param count to the full dense update. At what rank does LoRA stop being cheaper?

Forecast: at $r = 2048$ (half of 4096), is LoRA still smaller than dense?

LoRA count. $r (d + k) = 2048 \times 8192 = 16, 777, 216$ . Why this step? Plug into the same formula — nothing special except $r$ is now large.
Dense count. $d \times k = 4096 \times 4096 = 16, 777, 216$ . Why this step? This is the fixed size of a full $Δ W$ .
Compare. They are equal. At $r = 2048$ , LoRA uses exactly as many params as the full update. Why this step? Break-even: $r (d + k) = d k \Rightarrow r = \frac{d k}{d + k} = \frac{409 6 ^{2}}{8192} = 2048$ .
Conclusion. For $r > 2048$ , LoRA is more expensive than full fine-tuning — pointless.

Verify: break-even $r = \frac{d k}{d + k} = \frac{16 , 777 , 216}{8192} = 2048$ . ✓ This is the mathematical reason LoRA only helps when $r ≪ \frac{d k}{d + k}$ .

C5 — The $α / r$ scaling across ranks

Worked example Example 5 — keeping update strength fixed

You tuned well at $r = 8, α = 16$ . You want more capacity, so you switch to $r = 32$ . What $α$ keeps the effective update strength $\frac{α}{r}$ the same?

Forecast: does $α$ go up or down when you raise $r$ ?

Read the current scale. $\frac{α}{r} = \frac{16}{8} = 2$ . Why this step? $\frac{α}{r}$ is the actual multiplier hitting $B A$ ; that is what you must preserve.
Solve for new $α$ . Want $\frac{α _{new}}{32} = 2 \Rightarrow α_{new} = 64$ . Why this step? Bigger $r$ makes the raw $B A$ entries grow; dividing by the bigger $r$ compensates only if $α$ scales up too.
Interpretation. Capacity (how many directions the update can use) rose from 8 to 32, but strength (how hard it pushes) stayed at $2 \times$ . Why this step? $α / r$ decouples "capacity" from "magnitude" — you tune them independently.

Verify: $\frac{64}{32} = 2 = \frac{16}{8}$ . ✓ Same effective scale.

C6 — Initialization at step 0

C7 — Merge and zero-latency inference

Worked example Example 7 — collapsing to one matrix

After training, $W = [2003]$ , $B = [12]$ , $A = [0.5, - 1]$ , $r = 1$ , $α = 1$ . Compute the merged $W^{'} = W + \frac{α}{r} B A$ and confirm $W^{'} x$ equals the two-step LoRA output for $x = [11]$ .

Forecast: will the merged single matrix give the same answer as running $W$ and $B A$ separately?

Outer product $B A$ . $[12] [0.5, - 1] = [0.5 1 - 1 - 2]$ . Why this step? This dense $2 \times 2$ is the whole update, reconstructed from the two thin factors.
Merge. $W^{'} = [2003] + [0.5 1 - 1 - 2] = [2.5 1 - 1 1]$ . Why this step? $\frac{α}{r} = 1$ , so just add. Now there is one matrix — no extra layer at inference.
Check output. $W^{'} x = [2.5 - 1 1 + 1] = [1.5 2]$ .
Two-step output. $W x = (2, 3)$ ; $\frac{α}{r} B A x = [0.5 - 1 1 - 2] = (- 0.5, - 1)$ ; sum $= (1.5, 2)$ . Why this step? Same numbers ⇒ merging is exact, so inference latency is literally zero.

Verify: merged $W^{'} x = (1.5, 2)$ equals split-path $W x + \frac{α}{r} B A x = (1.5, 2)$ . ✓

C8 — Storage word problem (multi-task)

Worked example Example 8 — budgeting a real deployment

You serve one 7B base model (stored as 14 GB in 16-bit) plus LoRA adapters for 5 tasks, each 8 MB. Compare total storage against saving a full fine-tuned copy per task. Give the ratio.

Forecast: roughly how many GB does the PEFT approach save versus 5 full copies?

PEFT storage. $14, 000 MB + 5 \times 8 MB = 14, 040 MB$ . Why this step? One shared frozen base plus five tiny plug-ins.
Full-copy storage. $5 \times 14, 000 MB = 70, 000 MB$ . Why this step? Full fine-tuning writes a whole new 14 GB checkpoint per task.
Savings. $70, 000 - 14, 040 = 55, 960 MB \approx 54.6 GB saved$ .
Ratio. $70, 000/14, 040 \approx 4.99 \times$ smaller. Why this step? As you add more tasks the ratio approaches the number of tasks — the base cost is amortized.

Verify: PEFT $= 14040$ MB, full $= 70000$ MB, ratio $\approx 4.986$ . ✓

C9 — QLoRA memory exam twist

Worked example Example 9 — 4-bit base, 16-bit adapters

A 7B-parameter base loaded in 4-bit (0.5 bytes/param) plus LoRA adapters totalling 4.19M params in 16-bit (2 bytes/param). Estimate weight-memory. Compare with loading the base in 16-bit.

Forecast: does the tiny 16-bit adapter or the huge 4-bit base dominate memory?

4-bit base. $7 \times 1 0^{9} \times 0.5 B = 3.5 \times 1 0^{9} B = 3.5 GB$ . Why this step? 4 bits = 0.5 bytes; only the frozen weights get quantized (see Quantization).
16-bit adapters. $4.19 \times 1 0^{6} \times 2 B = 8.38 \times 1 0^{6} B \approx 8.4 MB$ . Why this step? Adapters stay high-precision so gradients are meaningful — but they're tiny, so this is negligible.
Total. $\approx 3.5 GB + 0.008 GB \approx 3.51 GB$ .
16-bit base for comparison. $7 \times 1 0^{9} \times 2 B = 14 GB$ . Why this step? Shows the 4-bit trick alone cuts weight memory $\approx 4 \times$ , letting huge models fit one GPU.

Verify: 4-bit base $= 3.5$ GB, adapters $\approx 0.00838$ GB, total $\approx 3.508$ GB; 16-bit base $= 14$ GB; ratio $= 4$ . ✓

C10 — Break-even rank exam twist

Worked example Example 10 — "at what rank is LoRA pointless?"

For a non-square weight $W \in R^{2048 \times 8192}$ , find the rank $r$ at which LoRA's parameter count equals the dense update. For which $r$ does LoRA actually save memory?

Forecast: will break-even $r$ be bigger or smaller than in the square case?

Set equal. $r (d + k) = d k \Rightarrow r = \frac{d k}{d + k}$ . Why this step? Break-even is where the two counts match — solve the formula for $r$ .
Plug in. $r = \frac{2048 \times 8192}{2048 + 8192} = \frac{16 , 777 , 216}{10 , 240} = 1638.4$ . Why this step? Direct substitution of the given dimensions.
Interpret. LoRA saves memory only for integer $r \leq 1638$ . Typical $r = 8$ – $16$ is vastly below this, so the savings are enormous. Why this step? Confirms why practical LoRA ranks are always deeply in the "cheap" region.

Verify: $\frac{2048 \times 8192}{2048 + 8192} = 1638.4$ . ✓ Since $8 ≪ 1638$ , LoRA is $\approx \frac{2048 \times 8192}{8 \times 10240} \approx 204.8 \times$ cheaper at $r = 8$ .

Active recall

Recall Which cell does each idea belong to?

Setting $r = 0$ gives what update? ::: The zero matrix — pure base model (Cell C3). The rank where LoRA stops saving memory is? ::: $r = \frac{d k}{d + k}$ (Cells C4, C10). Why does merging add zero latency? ::: $W^{'} = W + \frac{α}{r} B A$ is one matrix, same output (Cell C7). If you double $r$ , what keeps update strength fixed? ::: Double $α$ so $α / r$ is unchanged (Cell C5). In QLoRA which part is 4-bit? ::: Only the frozen base; adapters stay 16-bit (Cell C9).

Connections

Parameter-efficient fine-tuning (PEFT) — the parent this page drills.
Low-Rank Matrix Factorization — why $B A$ reconstructs a $d \times k$ update.
Full Fine-Tuning — the dense $Δ W$ these examples compare against.
Quantization — the 4-bit base in Example 9.
Transformer Attention — the $q, v$ projections these matrices usually are.
Prompt Tuning — a rank-free PEFT alternative.
Catastrophic Forgetting — why zero-init $B$ (Example 6) matters.