LoRA and QLoRA: Worked examples — AI-ML deep dive

Everything here builds on LoRA and QLoRA (parent). If a term is new, check Matrix Rank and Low-Rank Approximation, Quantization of Neural Networks, Adam Optimizer memory cost, or Full Fine-Tuning vs PEFT.

The scenario matrix

Every LoRA/QLoRA numeric question falls into one of these cells. We will hit each one.

Cell	What varies	The question it answers
C1 Param count, square	$d = k$	How many trainable params for one square projection?
C2 Param count, rectangular	$d \neq = k$	Does the formula still hold when shapes differ?
C3 Degenerate rank $r = 1$	smallest useful rank	What is the leanest LoRA possible?
C4 Limiting rank $r = min (d, k)$	rank saturates	When does LoRA stop saving anything?
C5 Scaling factor $α / r$	change $r$ , hold $α$	How does effective update strength move?
C6 Full-FT optimizer memory	Adam on all params	Why is full fine-tuning so heavy?
C7 QLoRA base storage	4-bit frozen weights	How small does the base get?
C8 Real-world sizing (word problem)	pick a GPU	Does a whole run fit in this card?
C9 Exam twist	mixed / trap	Apply LoRA to a subset of layers, count everything

Recall The three formulas everything below uses

Trainable params per matrix ::: $r (d + k)$ Effective update scale ::: $\frac{α}{r}$ Bytes/param at $b$ bits ::: $b /8$

C1 — Param count, square projection

C2 — Rectangular projection ( $d \neq = k$ )

C3 — Degenerate: the leanest possible rank ( $r = 1$ )

C4 — Limiting case: rank saturates ( $r = min (d, k)$ )

Worked example Cell C4

$d = k = 4096$ . What if someone sets $r = 4096$ — the maximum possible rank?

Forecast: at maximum rank, is LoRA still saving anything? Guess yes/no.

Compute LoRA params. $r (d + k) = 4096 (4096 + 4096) = 4096 \times 8192 = 33, 554, 432$ . Why this step? Push the formula to its limit to expose where it breaks even.
Compare to full. Full $= 409 6^{2} = 16, 777, 216$ . Why this step? We need the crossover.
Read the ratio. $33554432/16777216 = 2.0$ — LoRA now costs twice as much as just training $Δ W$ directly! Why this step? Because $B A$ stores two full-size matrices to represent what one matrix could hold. At full rank the factorization is pure overhead.

Verify: LoRA only saves when $r (d + k) < d k$ , i.e. $r < \frac{d k}{d + k}$ . For $d = k = 4096$ : $r < 2048$ . At $r = 4096$ we blew past that, so cost doubled. ✓ (This is why the whole method insists $r ≪ min (d, k)$ .)

C5 — Scaling factor $α / r$ across changes

C6 — Full fine-tuning optimizer memory (the baseline pain)

Worked example Cell C6

A $7$ B model, full fine-tuning with Adam in fp32. How much memory just for optimizer state + master weights + gradients?

Forecast: guess in GB before computing. (Hint: it is not the model's own 14 GB.)

Count Adam's copies. fp32 master weights (4 B) + gradient (4 B) + momentum $m$ (4 B) + variance $v$ (4 B) $= 16$ B/param. Why this step? Adam keeps two moving averages plus a master copy; see Adam Optimizer memory cost.
Multiply. $7 \times 1 0^{9} \times 16 = 1.12 \times 1 0^{11}$ bytes. Why this step? Total per-param cost times parameter count.
Convert to GB. $1.12 \times 1 0^{11} /1 0^{9} = 112$ GB (using $1 0^{9}$ bytes/GB). Why this step? Human-readable — and clearly won't fit a consumer GPU.

Verify: the parent quoted " $7 B \times 12 B \approx 84$ GB" using $12$ B/param (dropping the separate gradient copy). With the fuller $16$ B accounting we get $112$ GB. Either way it's tens of GB of pure optimizer overhead — the whole reason PEFT exists (Full Fine-Tuning vs PEFT). ✓

C7 — QLoRA 4-bit base storage

C8 — Real-world word problem: does it fit this GPU?

Worked example Cell C8 (word problem)

You have a single 24 GB GPU. You want to QLoRA-tune a $7$ B model with rank- $16$ adapters on the $Q$ and $V$ projections of $32$ layers, each projection $d = k = 4096$ . Add up base + adapters + adapter optimizer state. Does it fit?

Forecast: yes or no — and what dominates the budget?

Base (NF4). From C7: $3.5$ GB. Why this step? Frozen 4-bit base is the biggest single chunk.
Adapter params. Per matrix $r (d + k) = 16 \times 8192 = 131, 072$ . Matrices $= 2 (Q,V) \times 32 layers = 64$ . Total $= 131072 \times 64 = 8, 388, 608 \approx 8.39$ M. Why this step? Count all injected matrices, not just one (this is the exam trap — don't forget the layer multiplier).
Adapter memory (bf16 weights + Adam). Weights $2$ B + gradient $2$ B + $m, v$ in fp32 $4 + 4$ B $\approx 12$ B/param. $8.39 \times 1 0^{6} \times 12 = 1.01 \times 1 0^{8}$ B $\approx 0.10$ GB. Why this step? Only adapters carry optimizer state — this is the whole saving.
Sum. $3.5 + 0.10 \approx 3.6$ GB (before activations). Why this step? Compare to the 24 GB budget.

Verify: $3.6$ GB $≪ 24$ GB — fits with enormous headroom for activations. The base dominates, not the adapters. Contrast C6's $112$ GB for full FT: QLoRA turns an impossible run into a comfortable one. ✓

C9 — Exam twist: LoRA on a subset, count everything

Worked example Cell C9 (exam-style)

Twist: "A $13$ B model has $40$ layers. Rank- $8$ LoRA is applied to $Q, K, V, O$ (four projections, each $5120 \times 5120$ ) in only the top 20 layers. Give trainable params and the fraction of the full model." Watch the traps: subset of layers, four projections, $13$ B total.

Forecast: more or less than $1%$ of the model?

Per-matrix count. $r (d + k) = 8 (5120 + 5120) = 8 \times 10240 = 81, 920$ . Why this step? Base unit before multiplying by how many matrices we touch.
How many matrices? $4$ projections $\times 20$ layers $= 80$ matrices. Why this step? The trap: "top 20 of 40", not all 40 — halve the layer count.
Total trainable. $81920 \times 80 = 6, 553, 600 \approx 6.55$ M. Why this step? Sum over every touched matrix.
Fraction of full model. $6.5536 \times 1 0^{6} /13 \times 1 0^{9} = 5.04 \times 1 0^{- 4}$ . Why this step? Express as a share so it's comparable across models.

Verify: $5.04 \times 1 0^{- 4} = 0.0504% \approx 0.05%$ of the model is trainable — well under $1%$ , right in LoRA's usual $< 1%$ band. If a student forgot the "top 20" and used $40$ layers they'd get $13.1$ M ( $0.10%$ ) — double the correct answer. Read the fine print! ✓

Recall Quick self-test

LoRA stops saving memory when $r$ exceeds what value (for $d = k$ )? ::: $r = \frac{d k}{d + k} = \frac{d}{2}$ — beyond it, $r (d + k) > d k$ . In the C8 budget, which term dominates? ::: The 4-bit frozen base ( $3.5$ GB), not the adapters ( $0.1$ GB). Raising $r$ from 8 to 32 with fixed $α$ changes update strength by what factor? ::: $\times 0.25$ (quartered).