Parameter-efficient fine-tuning (PEFT): Question bank

Every symbol here comes from the parent note: $W$ is the frozen pretrained weight matrix, $Δ W = B A$ is the low-rank update ( $B$ is tall-thin $d \times r$ , $A$ is short-wide $r \times k$ ), $r$ is the rank knob, and $α / r$ is the scaling factor in $h = W x + \frac{α}{r} B A x$ .

True or false — justify

True or false: LoRA changes the pretrained weight matrix

W

during training.

False.

W

is frozen — gradients never touch it. Only

A

and

B

receive updates; the "change"

\frac{α}{r} B A

lives in separate matrices.

True or false: at training step 0, a LoRA model produces exactly the same outputs as the base model.

True. Because

B = 0

at init,

B A = 0

, so

h = W x + 0 = W x

— identical to the untouched base. Training only adds corrections from there.

True or false: increasing the rank

r

always improves fine-tuning quality.

False. Gains saturate around

r = 4

–

16

; larger

r

wastes memory and can overfit small data. LoRA works because the update is low rank — inflating

r

fights that premise.

True or false: QLoRA stores the trainable adapters in 4-bit.

False. Only the frozen base is 4-bit. Adapters stay in higher precision (e.g. 16-bit) so their gradients carry meaningful signal.

True or false: LoRA adds inference latency the same way adapters do.

False. Adapters insert extra sequential layers → real added compute per token. LoRA merges into

W^{'} = W + \frac{α}{r} B A

, leaving one matrix and zero overhead.

True or false: after fine-tuning with LoRA you must save the entire model to preserve the task.

False. The base is unchanged, so you save only

A, B

(a few MB) and reconstruct

W + \frac{α}{r} B A

on demand.

True or false: PEFT can reach roughly full fine-tuning quality while updating under 1% of parameters.

True. That is the core empirical claim — the useful weight change has low intrinsic dimension, so most of

W

's capacity is redundant for a single task.

True or false: freezing the base model helps prevent catastrophic forgetting.

True. Since prior knowledge lives in the untouched

W

, and

B A

starts at zero, early training cannot overwrite what the model already knew. See Catastrophic Forgetting.

True or false: the factorization

Δ W = B A

is only exact when

Δ W

actually has rank

\leq r

True. Any product

B A

with inner dimension

r

has rank at most

r

; if the true update needs higher rank,

B A

can only approximate it. See Low-Rank Matrix Factorization.

True or false: prompt tuning and LoRA are the same family of PEFT.

False. Prompt tuning is prompt-based (prepend trainable soft vectors to the input); LoRA is additive (inject a low-rank weight update). See Prompt Tuning.

Spot the error

Claim: "LoRA trains

d \cdot k

new parameters per adapted matrix."

Wrong — that's the size of a full

Δ W

. LoRA trains

r (d + k)

params (from

B : d \times r

plus

A : r \times k

), which is far smaller when

r ≪ d, k

Claim: "Set

B \sim N (0, σ^{2})

and

A = 0

— initialization symmetry means it doesn't matter which is zero."

Misleading. Only the product must start at zero. The parent uses

B = 0, A \sim N

; if both were random,

B A \neq = 0

and the model would jump away from the pretrained state at step 0, risking damage.

Claim: "To double capacity, raise

r

from 8 to 16 and keep

α = 16

Error — that changes the scale

α / r

from

2

1

, weakening the update. Keep the scale fixed by also setting

α = 32

α / r = 2

Claim: "QLoRA quantizes the base to save memory, so accuracy must drop a lot because the model computes in 4-bit."

Error in reasoning. The 4-bit weights are only read for the forward pass; the learning happens in full-precision adapters, so quality stays close to 16-bit LoRA. See Quantization.

Claim: "Merging

W^{'} = W + \frac{α}{r} B A

lets you keep swapping tasks at inference for free."

Error — once merged, the adapter is baked in and you lose easy hot-swapping. Multi-task serving keeps adapters unmerged so you can plug a different tiny

A, B

per request.

Claim: "BitFit is an additive PEFT method that inserts bottleneck MLPs."

Wrong on two counts. BitFit is selective (trains only existing bias terms); bottleneck MLPs are adapters, a different additive method.

Why questions

Why store

Δ W

as two thin matrices instead of one dense matrix?

Because the task update has low intrinsic dimension —

r (d + k)

numbers capture it, versus

d \cdot k

for a dense update, e.g. 256× fewer at

d = k = 4096, r = 8

Why divide by

r

in the scaling factor

α / r

The typical magnitude of

B A

's entries grows with

r

; dividing by

r

decouples how strong the update is (

α

) from how much capacity it has (

r

), so retuning

r

doesn't blow up the update.

Why is LoRA usually applied to the

q

and

v

projections of Transformer Attention?

These projections shape what the model attends to and retrieves; empirically, low-rank updates there recover most task accuracy at minimal cost.

Why does full fine-tuning need ~3× the memory of just the weights?

You also store gradients for every parameter plus optimizer state — Adam keeps two moments — roughly doubling on top of the weights themselves.

Why can you serve many tasks from one base with PEFT cheaply?

The giant base stays in memory once; each task is a few-MB adapter you hot-swap per request, turning "one model per task" into "one model, many plug-ins."

Why does gradient flow through the frozen 4-bit weights in QLoRA even though they aren't updated?

The forward output depends on those weights, so backprop must pass through them to reach the adapters — but the base's own gradient is discarded since it's frozen.

Edge cases

What happens if you set

r = 0

Then

B, A

have no columns/rows to train —

B A

is identically zero forever, so

h = W x

and the model can never adapt. It degenerates to using the frozen base as-is.

What happens if

r = min (d, k)

B A

can represent a full-rank

Δ W

, so you regain the expressive power of a dense update — but you also lose the parameter savings, defeating the purpose of LoRA.

α = 0

, what does the LoRA layer compute?

h = W x + 0 = W x

— the update is switched off entirely, equivalent to running the frozen base regardless of what

A, B

learned.

Two adapters trained separately for tasks X and Y — can you just add both

B A

terms to serve both at once?

Not reliably.

\frac{α}{r} (B_{X} A_{X} + B_{Y} A_{Y})

is an untrained sum of independent updates; nothing guarantees the tasks compose linearly, so you may get interference rather than both behaviours.

If the true task update genuinely needs high rank, what does LoRA lose?

B A

can only approximate a high-rank

Δ W

, so accuracy falls short of full fine-tuning — this is the regime where PEFT's low-rank assumption breaks down.

After merging, can you recover the original base weights

W

Yes, if you kept

A, B

: compute

W = W^{'} - \frac{α}{r} B A

. If you discarded the adapters, the subtraction is impossible and

W

is lost inside

W^{'}

Connections

Full Fine-Tuning — the baseline whose memory cost these traps compare against.
Low-Rank Matrix Factorization — why $Δ W = B A$ caps rank at $r$ .
Quantization — the 4-bit read-only trick behind QLoRA.
Transformer Attention — where $q, v$ LoRA lives.
Prompt Tuning — the prompt-based sibling family.
Catastrophic Forgetting — mitigated by freezing + zero-init.