LoRA and QLoRA

Parameter-efficient fine-tuning: adapt a huge frozen model by training a tiny number of extra parameters.

The core problem

WHY we need this: cost, storage (one adapter per task = few MB vs full model copies), and speed. WHAT LoRA does: freezes original weights, injects a trainable low-rank correction. HOW QLoRA extends it: it also quantizes the frozen base to 4-bit so it fits on a small GPU.

Deriving LoRA from scratch

A linear layer computes $h = W_0 x$ where $W_0 \in \mathbb{R}^{d\times k}$ .

Fine-tuning wants a new weight $W = W_0 + \Delta W$ . The full update $\Delta W$ has $d\times k$ parameters — as many as $W_0$ .

Why this factorization? Any matrix of rank $r$ can be written as a sum of $r$ outer products. $BA$ is exactly that: $r$ columns of $B$ times $r$ rows of $A$ . So $BA$ can represent any rank- $\le r$ matrix, and nothing more — that's the whole point (we deliberately restrict ourselves).

Parameter count: instead of $d\cdot k$ , we train only $r(d+k)$ . For $d=k=4096$ , $r=8$ : full $= 16.7$ M vs LoRA $= 65\text{k}$ — a 256× reduction.

Why $B=0$ at init? If both $A,B$ were random, the model output would be corrupted at step 0. Starting with $\Delta W = 0$ means the first gradient step is a pure improvement direction.

Why divide by $r$ ? As you raise $r$ , the sum $BAx$ accumulates more terms and grows. Dividing by $r$ normalizes so you don't have to re-tune the learning rate every time you change rank.

The magic at inference

After training you can merge: $W \leftarrow W_0 + \frac{\alpha}{r}BA$ . Now it's a normal weight matrix — zero extra latency. This is LoRA's killer feature vs adapter layers (which add compute).

QLoRA = Quantization + LoRA

QLoRA's goal: fine-tune a 65B model on a single 48GB GPU. Three ingredients:

The key trick: the base weights are stored in 4-bit (frozen). During the forward pass they are dequantized to bf16 on the fly just for the matmul. Only the LoRA matrices $A,B$ are kept in full precision and receive gradients.

Why quantizing the frozen base is safe: we're not backpropagating into it. Quantization noise on frozen weights is a fixed, small perturbation the adapters learn to compensate for. You'd never quantize weights you're trying to train precisely.

Worked examples

Common mistakes (steel-manned)

Flashcards

What low-rank factorization does LoRA use for the weight update?

\Delta W \approx BA

with

B\in\mathbb{R}^{d\times r}

A\in\mathbb{R}^{r\times k}

r\ll\min(d,k)

How many parameters does rank-

r

LoRA train per

d\times k

matrix?

r(d+k)

instead of

d\cdot k

Why is

B

initialized to zero?

\Delta W = BA = 0

at start; training begins exactly at the pretrained model with no output corruption.

Why divide by

r

\frac{\alpha}{r}BA

Keeps update magnitude roughly independent of rank, so learning rate need not be retuned per rank.

What is the inference-latency cost of LoRA after merging?

Zero — merge

W_0+\frac{\alpha}{r}BA

into one matrix; it's a normal linear layer.

What are QLoRA's three innovations?

4-bit NormalFloat (NF4), Double Quantization, Paged Optimizers.

In QLoRA, which parts are quantized and which receive gradients?

The frozen base weights are 4-bit; only the full-precision LoRA adapters

A,B

receive gradients.

Why is quantizing the frozen base safe?

No gradients flow into it; adapters learn to compensate for the fixed quantization noise.

What is NF4 optimal for and why?

Normally-distributed data; its bins are quantiles of a normal distribution, matching the weight distribution.

Rule of thumb linking

\alpha

and

r

Set

\alpha \approx 2r

to keep effective update strength consistent.

Recall Feynman: explain to a 12-year-old

Imagine a huge finished LEGO castle (the pretrained model). You want to change it for a birthday party, but rebuilding the whole castle is crazy expensive. Instead you add a tiny set of removable decorations (the small $A,B$ matrices) — cheap to make, easy to store, and you can swap them for different parties. LoRA = tiny decorations. QLoRA = you also shrink the original castle bricks (4-bit) so the whole thing fits on your small table, while still decorating with high-quality add-ons.

Connections

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek bada LLM (jaise 7B parameters) ko poora fine-tune karna bahut mehenga hai — Adam optimizer ke saath ~80GB memory chahiye, sirf ek task ke liye. LoRA ka idea simple hai: original weights ko freeze kar do, aur weight ka jo change ( $\Delta W$ ) chahiye usko ek chhoti si low-rank form mein likho: $\Delta W = BA$ , jahan $B$ aur $A$ patli matrices hain (rank $r$ bahut chhota). Isse parameters $d\times k$ se ghat kar $r(d+k)$ ho jaate hain — 100x se zyada kami. $B$ ko zero se init karte hain taaki shuru mein model bilkul pretrained jaisa hi behave kare, aur training smoothly improve kare.

QLoRA ek step aage jaata hai: base weights ko 4-bit NF4 mein store karta hai (memory aur kam), lekin train sirf full-precision LoRA adapters hote hain. Forward pass ke time base ko on-the-fly bf16 mein dequantize karke matmul karte hain. Kyunki base pe gradient nahi jaata, 4-bit ki quantization noise se koi bada nuksaan nahi — adapters us noise ko compensate karna seekh lete hain. Isliye 65B model bhi ek single GPU pe fine-tune ho jaata hai.

Ek important baat: $\alpha/r$ scaling. Agar rank badhao par $\alpha$ same rakho, to effective update strength kam ho jaati hai — isliye rule of thumb: $\alpha \approx 2r$ . Aur inference pe LoRA free hai kyunki tum $W_0 + \frac{\alpha}{r}BA$ ko merge karke ek normal matrix bana sakte ho — koi extra latency nahi. Yeh sab isliye kaam karta hai kyunki task-specific adaptation genuinely low-dimensional hoti hai.

The core problem

Deriving LoRA from scratch

The magic at inference

QLoRA = Quantization + LoRA

Worked examples

Common mistakes (steel-manned)

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Pretraining & Fine-Tuning LLMs

Connections