Exercises — Full fine-tuning vs feature extraction

• $x$ ::: one input example (a sentence, an image).
• $f_{\theta }(x)$ ::: the backbone — the big pretrained network. It turns $x$ into a feature vector. $\theta$ is its pile of weights.
• $h_{\varphi }(\cdot )$ ::: the head — a small new layer on top. $\varphi$ is its (few) weights.
• $\hat{y}=h_{\varphi }(f_{\theta }(x))$ ::: the model's prediction.
• $\mathcal{L}$ ::: the loss — a number saying how wrong $\hat{y}$ is.
• $\eta$ ::: learning rate — how big a step gradient descent takes.
• $P_{\text{backbone}},P_{\text{head}}$ ::: how many numbers live in $\theta$ and $\varphi$.
• "freeze" ::: mark params as requires_grad=False so no gradient is applied to them.

Only $==\varphi ==$ (the head). The backbone $\theta$ is frozen — its gradient $\partial \mathcal{L}/\partial \theta$ is discarded (or never computed), so $\theta$ never moves.

Fewest → feature extraction (linear probing): train only the head. Most → full fine-tuning: train backbone + head. Everything else (PEFT/LoRA, partial unfreezing, discriminative LRs) sits between them.

True. Because $\theta$ is frozen, $f_{\theta }(x)$ is a constant vector for each $x$. Compute it once, cache it, and train the head on the cached vectors — no repeated backbone forward passes.

A linear layer from $768\to 3$ has a weight matrix $768\times 3$ plus $3$ biases: $768\times 3+3=2304+3===2307==$ That is the entire trainable count in feature extraction.

Full FT trains everything: $P_{\text{full}}=110,000,000+2307=110,002,307$ Ratio to feature extraction: $\frac{110,002,307}{2307}\approx 47,681$ Full FT trains roughly 47 thousand times more parameters. This is why memory and compute blow up.

$\text{Mem}\approx 4\times P_{\text{trainable}}\times 4\text{bytes}=16\times 110,000,000\text{bytes}$ $=1,760,000,000\text{bytes}===1.76\text{GB}==$ Compare feature extraction: $16\times 2307\approx 36,912$ bytes $\approx 0.000037$ GB — effectively nothing. This is the "full FT is heavy" term from the parent note. See Learning Rate Schedules for how $\eta$ interacts with this cost.

With only 500 examples, full FT's ~110M free params vastly outnumber the data → the model has huge variance and will fit noise (overfit). Since the task is close to the pretraining domain, the frozen features already encode what you need → bias is low even without adapting $\theta$. Pick feature extraction. Few trainable params act as strong regularization; the domain closeness means you lose almost nothing by freezing.

In a deep net, shallow (bottom) layers learn general features (edges, generic token statistics) that transfer across domains; deep (top) layers learn task/domain-specific combinations. A 71→79% jump from unfreezing only the top 2 layers means the mismatch was in the specific features, exactly where the top layers live. Keeping the bottom frozen:

• preserves transferable knowledge (avoids Catastrophic Forgetting),
• trains far fewer params than full FT (cheaper, less overfitting). This is the partial fine-tuning middle of the spectrum.

• $\frac{\partial \mathcal{L}}{\partial \hat{y}}$ — still computed (needed to train the head).
• $\frac{\partial h_{\varphi }}{\partial f_{\theta }}$ — this factor would carry the signal into the backbone. In feature extraction you don't need to push it further because...
• $\frac{\partial f_{\theta }}{\partial \theta }$ — never needed / never computed. Since $\theta$ is frozen, there is no reason to backpropagate through the (huge) backbone.

Meaning: the backward pass stops at the boundary between head and backbone. No gradient flows through the expensive $f_{\theta }$ internals — that is precisely why feature extraction is cheap.

Serving 50 full-FT copies means 50 × (backbone size) on disk/GPU — infeasible. Instead:

• Freeze the shared backbone once (feature extraction style) → one copy of $\theta$ serves all tasks.
• Give each task its own tiny head (or a LoRA/adapter module of a few million params).
• At inference, load the shared backbone plus the small per-task module.

Budget: trainable params per task = head/adapter only (KB–MB), not the ~110M backbone. This is the "limited GPU / many tasks" row of the parent's decision table — feature extraction or PEFT, never 50 full fine-tunes.

• Unfreeze everything. 200k examples + big domain shift means frozen features are biased (wrong for legalese); there's enough data to justify full capacity.
• Small LR, $\eta \approx 2\times 10^{-5}$. The model already sits near a good optimum; a large step would overwrite pretrained knowledge → catastrophic forgetting. Small $\eta$ nudges features gently.
• Warmup (see Learning Rate Schedules): start $\eta$ near 0 and ramp up. Early gradients through deep layers can be large and noisy; warmup stops them from wrecking the initialization.
• Weight decay: mild regularization that keeps weights from drifting too far from their (good) pretrained values.

Result: the backbone reshapes toward legalese while retaining general language competence — beats feature extraction because the data justifies the extra capacity.

Order experiments by increasing trainable params (cheapest, safest first):

1. Feature extraction (baseline accuracy $a_0$).
2. Unfreeze top 1 layer → $a_1$. Top 2 → $a_2$. Continue while each added layer gives a worthwhile accuracy jump.
3. Full FT only if the curve $a_k$ is still climbing at the top layers (signals deep, pervasive domain shift).

Stopping criterion: stop unfreezing when the marginal accuracy gain $a_k-a_{k-1}$ falls below your tolerance, or when validation accuracy starts dropping (overfitting from too many free params on your data size). This walks the spectrum from cheap/robust toward expensive/flexible, spending compute only where it pays.

• Student A: $\theta$ frozen → $f_{\theta }(x)$ is identical to the pretrained model's, unchanged.
• Student B: applies $\theta \leftarrow \theta -\eta \partial \mathcal{L}/\partial \theta$ each step → $\theta$ changes, so $f_{\theta }(x)$ evolves and ends up different from the pretrained features.

Prediction: the two feature vectors differ. This kills the myth "both give the same features." Only full FT adapts internal representations; feature extraction keeps them fixed by construction.

(a) Trainable $\approx 110,002,307$. $16\times 110,002,307\approx 1.76\times 10^9\text{bytes}\approx ==1760\text{MB}==$ (b) Trainable $=1,200,000$. $16\times 1,200,000=1.92\times 10^7\text{bytes}===19.2\text{MB}==$ Reduction factor: $1760/19.2\approx ==91.7\times ==$ less optimizer memory. This is why PEFT and LoRA dominates when GPUs are scarce.

• Feature extraction trains ::: only the head; backbone frozen; features cached.
• Full FT trains ::: backbone + head; features evolve; heavy memory.
• Small data → prefer ::: feature extraction (low variance).
• Large data + domain shift → prefer ::: full fine-tuning (low bias).
• Many tasks / low GPU → prefer ::: feature extraction or PEFT/LoRA.
• Small LR + warmup in full FT to ::: avoid catastrophic forgetting.