Question bank — LoRA and QLoRA

False. After merging, $W_0+\frac{\alpha }{r}BA$ is one full-size $d\times k$ matrix (output dim $d$, input dim $k$) — same size as before. LoRA reduces trainable parameters during fine-tuning, not the model's footprint.

False. It injects a low-rank correction into an existing $d\times k$ linear layer's weights; unlike Adapter Layers there are no new sequential blocks, and after merging there's literally no trace left.

False. $BA$ (with $B$ of shape $d\times r$ and $A$ of shape $r\times k$) has rank at most $r\ll \min (d,k)$, so a full-rank update is fundamentally outside its reach — LoRA bets the needed update is low-rank.

False. Frozen means unchanged and fully used on every forward pass; the knowledge is preserved perfectly — only a small correction is added on top.

False. Only the frozen base is stored in 4-bit; the trainable adapters $A,B$ stay full precision and the base is dequantized to bf16 for each matmul. Nothing that receives a gradient is 4-bit.

True — by definition $\frac{\alpha }{r}=\frac{r}{r}=1$ for every $r$, so the scale is exactly $1$ regardless of rank. The separate heuristic $\alpha \approx 2r$ instead fixes the scale at $2$; don't conflate "is the scale constant?" (yes for $\alpha =r$) with "which constant should I pick?" (a tuning question).

False. $B=0$ only makes $\Delta W=0$ at step 0. Gradients flow into $B$ through $A$ (which is nonzero), so $B$ immediately moves away from zero on the first update.

True in spirit — either choice gives $BA=0$ initially. The convention is $B=0$, $A$ Gaussian; the key requirement is just that the product starts at zero.

False. Merging gives zero latency for one adapter, but if you serve many task adapters on the same base simultaneously you must keep them unmerged to swap them cheaply.

The 14 GB is just the fp16 weights. Full FT with Adam also needs momentum + variance + fp32 master weights (~12 bytes/param) → ~84 GB. The trap is quoting only weight storage.

There are no gradients for the base at all — it's frozen. Gradients exist only for $A,B$; the whole point is you never need high-precision gradients for a base you don't update.

Both are 4 bits. NF4 wins by placing its 16 levels at the quantiles of a normal distribution, matching where weight values actually cluster — same bit budget, smarter bin placement.

$\alpha$ is a scaling scalar, not capacity. Capacity comes from rank $r$ (the width of the waist in $BA$, i.e. the number of independent directions). Doubling $\alpha$ just scales the same low-rank update up.

The reason it overfits less isn't fewer parameters per se — it's that the low-rank constraint restricts the update to a genuinely low-dimensional, task-relevant subspace, blocking noisy high-rank directions.

More placement can help but also adds parameters and can hurt; papers often find Q,V alone is a strong sweet spot. It's a tradeoff, not a monotone "more is better."

No — it quantizes the quantization constants (the per-block scales of the frozen base), saving ~0.37 bits/param. The adapters are never quantized.

As $r$ grows, $BAx$ sums more outer-product terms and tends to grow. Tiny numeric sketch: if each term contributes ~magnitude $1$, then rank $r=4$ gives ~$4$ but rank $r=16$ gives ~$16$; dividing by $r$ pulls both back to ~$1$, so the learning rate stays valid across ranks.

On a frozen weight the 4-bit rounding error $ϵ$ is a single fixed offset the adapters cancel. But in training you round every step, and gradient descent accumulates those errors: over $N$ steps the drift grows like $N\cdot ϵ$ instead of staying $ϵ$ — see the diverging red curve.

LoRA's correction is linear in the same layer and merges into one matrix; adapter layers are extra nonlinear blocks in series that must be computed every forward pass and can't be folded away.

At step 0 the model output equals the pretrained model exactly, so the loss gradient points purely toward task improvement — not toward undoing random corruption you injected.

Adapting to one narrow task nudges the model along a few task-relevant directions; the base already encodes general knowledge, so the change needed is a small, structured perturbation, not a full-rank rewrite.

They only relocate optimizer state to CPU RAM during transient GPU memory spikes (e.g., a long sequence), preventing an out-of-memory crash — a memory-management safety net, not an algorithmic change.

$BA$ vanishes ($B$ has shape $d\times 0$, $A$ has shape $0\times k$) → $\Delta W=0$ permanently. You've frozen the whole model; it's inference-only, no adaptation possible.

$BA$ can represent an arbitrary full update, so you recover something like full fine-tuning of that layer — but you've thrown away the parameter savings and re-invited overfitting.

No — standard LoRA adapts only the weight matrix $W_0$ (shape $d\times k$); the bias $b$ (length $d$) stays frozen and unchanged. If a task truly needs a shifted bias you must unfreeze $b$ explicitly (it's cheap: only $d$ numbers), but the vanilla recipe leaves it alone, which is why merging is a clean weight-only fold.

No — once merged, $\Delta W$ is baked in and can't be cleanly removed. Keep the base + separate adapters if you need to swap tasks.

Mathematically you can sum them, but they were each trained assuming the other wasn't present, so their interaction isn't accounted for; results can degrade — it's an approximation, not a guarantee.

Less so. NF4's optimality assumes near-normal weights; for a very different distribution its quantile bins are mismatched and a distribution-matched or uniform scheme could do better.

Yes — every forward pass (including step 0) dequantizes the frozen NF4 base to bf16 for the matmul; the 4-bit form is purely a storage format, not a compute format.