Worked examples — Full fine-tuning vs feature extraction

This page is the exercise gym for the parent note. There we built the two anchor strategies. Here we walk every case class the decision can throw at you — tiny data, huge data, domain shift, memory limits, degenerate inputs — and work each one fully.

Before we start, one reminder of the notation so nobody is lost from line one:

---

The scenario matrix

Every real fine-tuning decision is a combination of a few axes. Think of the matrix below as "all the dials". Each worked example lights up specific cells so that, together, they cover every cell at least once.

Axis	Extreme A	Middle	Extreme B
Dataset size	tiny (hundreds)	medium (thousands)	huge (100k+)
Domain shift vs pretraining	none / close	some	large / "legalese"
Trainable params	head only (feat. extr.)	top-$k$ layers	everything (full FT)
Compute budget	1 small GPU	1 big GPU	cluster
Degenerate inputs	$0$ data or $0$ trainable params	LR $=0$ (no step)	LR huge (overshoot)

The "degenerate" row is the equivalent of checking every sign and the zero case in a geometry problem: what happens at the boundaries where a formula could blow up or silently do nothing. Its Middle column is not "less degenerate" — it is the intermediate boundary where a step is taken but its size is exactly $0$, sitting between "nothing to train" (left) and "step too big" (right). We hit all three columns in Examples 5–7 and 10.

The picture below is the same matrix as a map — read it top to bottom to pick a strategy:

Coverage map (fill it in as you read):

Cell	Example
Tiny data · close domain · head only	Ex 1
Huge data · big shift · full FT	Ex 2
Medium data · some shift · top-$k$	Ex 3
Memory math (why full FT is heavy)	Ex 4
Degenerate: LR $=0$ (Middle column)	Ex 5
Degenerate: LR far too large (right)	Ex 6
Degenerate: $0$ trainable params (left)	Ex 7
Real-world word problem: serving many tasks	Ex 8
Exam twist: forgetting curve limit	Ex 9
Degenerate: $0$ data (left)	Ex 10

---

Example 1 — Tiny data, close domain (head only)

1. Count head params. A linear layer from $768$ inputs to $2$ outputs has a weight matrix of $768\times 2$ plus one bias per output: $P_{\text{head}}=768\times 2+2=1538.$ Why this step? Trainable-parameter count is our proxy for overfitting risk. $1538$ on $500$ examples is modest — good.
2. Precompute embeddings once. Because $\theta$ is frozen, $f_{\theta }(x)$ never changes, so run the backbone once per review: $500$ forward passes total, cached forever. Why this step? A frozen function of a fixed input is a constant — recomputing it every epoch would be pure waste. This is the Transfer Learning free lunch.
3. Train the head fast. Logistic regression on $500$ cached $768$-vectors trains in seconds. Why this step? No backward pass through the backbone means near-zero cost.

Verify: $768\times 2+2=1538$. Backbone forward passes $=500$ (once each), backbone backward passes $=0$. Sanity on data-vs-params: here we train more parameters ($1538$) than we have examples ($500$) — ratio $500/1538\approx 0.33$ examples per parameter. That is under-determined, which is exactly why we lean on a linear head (a highly constrained shape) plus regularization; if we had instead unfrozen the whole $110$M-param backbone the mismatch would be catastrophic, so the Bias-Variance Tradeoff says freeze. ✓

---

Example 2 — Huge data, big domain shift (full fine-tuning)

1. Diagnose the regime. 200k examples is plenty to constrain a big model → variance is not the bottleneck. But frozen generic features cannot represent "legalese" → high bias. Why this step? The Bias-Variance Tradeoff tells you which knob to turn: here, reduce bias by giving the model capacity to reshape features.
2. Unfreeze $\theta$, use a small $\eta$. Update both, pushing each weight downhill on the loss $\mathcal{L}$: $\theta \leftarrow \theta -\eta \frac{\partial \mathcal{L}}{\partial \theta },\varphi \leftarrow \varphi -\eta \frac{\partial \mathcal{L}}{\partial \varphi }.$ Why a small $\eta$? The model already sits near a good optimum; a large step would overwrite good features — Catastrophic Forgetting. Small steps nudge rather than bulldoze.
3. Add warmup + weight decay. Start $\eta$ near $0$ and ramp up over the first few hundred steps. Why this step? Early gradients through deep layers can be large; warmup (Learning Rate Schedules) prevents a destructive first jump.

Verify: $\eta =2\times 10^{-5}=0.00002$, which sits in the recommended full-FT band $[10^{-5},10^{-4}]$. Data-to-shift logic: huge data ⇒ variance safe ⇒ full FT justified. No arithmetic to fail here, but the LR is inside band: $10^{-5}\le 2\times 10^{-5}\le 10^{-4}$. ✓

---

Example 3 — Medium data, some shift (top-$k$ partial fine-tuning)

1. Recall the layer hierarchy. In Layer-wise Representations in Deep Nets, shallow layers learn generic patterns; deep (later) layers learn task/domain-specific ones. Why this step? It tells you where to spend your limited retraining budget.
2. Unfreeze the top 2 layers only. Now $P_{\text{trainable}}=P_{\text{top 2}}+P_{\text{head}}$ — far less than full FT. Why this step? You adapt the specific part while protecting the general part from forgetting.
3. Read the payoff. Accuracy climbs $71\%\to 79\%$. Why this step? This is a point on the spectrum between the two anchors — cheaper than full FT, more flexible than pure feature extraction.

Verify: improvement $=79-71=8$ percentage points. Fraction of trainable layers strictly between the two anchors ($0<\text{top-}k<\text{all}$), confirming it is a genuine middle point. ✓

---

Example 4 — Memory math: why full FT is heavy

1. Full FT trainable count. $P_{\text{trainable}}^{\text{full}}=110,000,000+1538=110,001,538.$ Why this step? Memory scales with the number of things Adam must track.
2. Full FT memory. Using $\text{Mem}\approx 4\times P_{\text{trainable}}\times 4\text{bytes}$: ${\text{Mem}}_{\text{full}}\approx 16\times 110,001,538\text{bytes}\approx 1.76\times 10^9\text{bytes}\approx 1.76\text{GB}.$ Why this step? $4$ numbers $\times$ $4$ bytes $=16$ bytes per trainable param — the constant that makes full FT painful.
3. Feature-extraction memory. ${\text{Mem}}_{\text{feat}}\approx 16\times 1538\approx 24,608\text{bytes}\approx 0.0000246\text{GB}.$ Why this step? Only the head is trainable, so the optimizer footprint nearly vanishes — you pay just the forward pass on the backbone.

Verify: ratio $\frac{{\text{Mem}}_{\text{full}}}{{\text{Mem}}_{\text{feat}}}=\frac{110,001,538}{1538}\approx 71,522\times$. Full FT's optimizer state is ~$71,000$ times heavier. Units: bytes $\to$ GB by dividing by $10^9$. ✓ (PEFT and LoRA exists precisely to kill this term.)

---

Example 5 — Degenerate: learning rate = 0

1. Apply the update rule. For any weight $w$: $w\leftarrow w-0\cdot \frac{\partial \mathcal{L}}{\partial w}=w.$ Why this step? Multiplying the gradient by $0$ erases the step no matter how large the gradient — the compass still points, but you take a step of length zero.
2. Interpret. $\theta$ and $\varphi$ both stay fixed → nothing learns, not even the head. Why this step? This is worse than feature extraction: feature extraction freezes $\theta$ but still trains $\varphi$ with a real $\eta$. Zero LR freezes everything.

Verify: with $\eta =0$, $w_{\text{new}}=w_{\text{old}}$ exactly for both $\theta$ and $\varphi$. So the model output equals the untrained head's output — no learning occurs. ✓ Boundary confirmed: $\eta =0$ ≠ feature extraction.

---

Example 6 — Degenerate: learning rate far too large

1. Compare to the safe band. Recommended full-FT range is $[10^{-5},10^{-4}]$. $\frac{10^{-1}}{10^{-4}}=10^3.$ Why this step? Quantifies just how far outside safe territory you are — $1000\times$ the top of the band.
2. Predict the damage. A step $1000\times$ too big overshoots the good optimum and overwrites pretrained features → severe Catastrophic Forgetting. Why this step? The model was near a minimum; a giant step throws it out of the basin entirely.

Verify: $10^{-1}/10^{-4}=1000$, and $10^{-1}>10^{-4}$, so $\eta =0.1$ is $1000\times$ above the safe ceiling. Clearly out of band. ✓ Fix: use $10^{-5}$–$10^{-4}$ or discriminative LRs.

---

Example 7 — Degenerate: no head, backbone frozen

1. Count trainable params. $P_{\text{trainable}}=P_{\text{backbone}}^{\text{(frozen)}}+P_{\text{head}}^{\text{(none)}}=0+0=0.$ Why this step? If nothing is trainable, gradient descent has nothing to move.
2. Interpret. The pipeline is a pure fixed function $x\mapsto f_{\theta }(x)$. It computes embeddings but cannot map them to your labels. Why this step? This is not even feature extraction — feature extraction needs the head ($P_{\text{head}}>0$) to learn the label mapping.
3. Fix. Add a trainable head: at minimum a linear layer, giving $P_{\text{head}}=d\cdot C+C$ for $d$ features and $C$ classes. Why this step? Restores a nonzero trainable set so learning can happen.

Verify: with $P_{\text{trainable}}=0$, no weight satisfies $w\leftarrow w-\eta \nabla$ with a nonzero move, so accuracy stays at the random/fixed-mapping baseline. For $d=768,C=2$ the minimal fix gives $768\cdot 2+2=1538$ trainable params. ✓

---

Example 8 — Real-world word problem: serving 12 tasks on one GPU

1. Full FT cost. 12 separate fine-tuned models: $12\times 110,000,000=1,320,000,000$ params to store. Why this step? Full FT changes $\theta$, so each task needs its own full weight set — unaffordable.
2. Feature extraction / PEFT cost. Share one frozen backbone; store only per-task heads. With $1538$-param heads: $12\times 1538=18,456$ trainable params total. Why this step? A frozen backbone is task-agnostic, so it is stored once; only tiny heads differ. See PEFT and LoRA for the adapter version.
3. Decide. Close domain + tiny per-task data + shared serving ⇒ feature extraction (or PEFT) wins on every axis. Why this step? Matches the matrix: small data → freeze; many tasks → share backbone.

Verify: storage ratio $\frac{12\times 110,000,000}{110,000,000+12\times 1538}\approx 12.0$ vs $\approx 1.0002$. Feature extraction stores essentially one backbone, so ~$12\times$ cheaper. ✓

---

Example 9 — Exam twist: the forgetting limit

1. Limit $\eta \to 0$. $R(0)=e^0=1.$ Why this step? No step ⇒ no overwrite ⇒ perfect retention. This is the feature-extraction / frozen extreme: nothing is nudged, so nothing pretrained is lost.
2. Safe LR $\eta =2\times 10^{-5}$. $R=e^{-5000\cdot 2\times 10^{-5}}=e^{-0.1}\approx \mathrm{0.905.}$ Why this step? We plug the actual recommended LR into the retention formula to see what "safe" buys you: about $90\%$ of pretrained knowledge survives while the model still adapts. This is precisely why Example 2 chose $2\times 10^{-5}$ — small enough to remember, nonzero enough to learn.
3. Limit $\eta \to \infty$. ${\lim }_{\eta \to \infty }{e}^{-k\eta }=0.$ Why this step? We check the far boundary: an unbounded step size drives retention to $0$ — total Catastrophic Forgetting, the exact failure mode of Example 6's oversized LR. Covering both limits ($0$ and $\infty$) shows the whole curve, not just one end.

Verify: $R(0)=1$; $R(2\times 10^{-5})=e^{-0.1}\approx 0.9048$; ${\lim }_{\eta \to \infty }R=0$. Monotonic decreasing, matching intuition "bigger LR forgets more". ✓

---

Example 10 — Degenerate: zero training data

1. Look at where gradients come from. The loss $\mathcal{L}$ is an average over training examples. With $0$ examples there is nothing to average, so $\mathcal{L}$ is undefined and no gradient $\frac{\partial \mathcal{L}}{\partial w}$ can be formed. Why this step? The whole update rule $w\leftarrow w-\eta \frac{\partial \mathcal{L}}{\partial w}$ needs a gradient built from data; no data ⇒ no gradient ⇒ no legitimate step for either $\theta$ or $\varphi$.
2. Interpret both strategies. Feature extraction: the head $\varphi$ stays at its random initialization → predictions are chance-level. Full FT: $\theta$ also never moves → the model is exactly the pretrained model with a random head. Why this step? Zero data collapses both anchor strategies to "no learning", regardless of what you froze — the data axis dominates.
3. Contrast with $\eta =0$ (Ex 5). Ex 5 had data but a zero step size; here we have a nonzero $\eta$ but no data. Both give "no learning", but for different reasons — one kills the step, the other kills the gradient. Why this step? Distinguishing the two zero-cases prevents the mistake of thinking "the model didn't learn" always means "bad learning rate".

Verify: number of gradient terms $=$ number of examples $=0$, so the summed/averaged gradient is empty and every $w$ satisfies $w_{\text{new}}=w_{\text{old}}$. A binary head at random init sits at chance accuracy $=1/C=1/2=0.5$. ✓

---

Recall Quick self-test

• Why can Example 1 cache embeddings but Example 2 cannot? ::: Ex 1 freezes $\theta$ so $f_{\theta }(x)$ is constant; Ex 2 changes $\theta$ every step.
• What makes $\eta =0$ worse than feature extraction? ::: Zero LR freezes the head too, so nothing learns; feature extraction still trains the head.
• How is "zero data" different from "$\eta =0$"? ::: Zero data means no gradient can be formed at all; $\eta =0$ means the gradient exists but the step length is $0$.
• Why is full FT ~$71,000\times$ heavier in optimizer memory here? ::: It tracks $110\text{M}+1538$ params vs $1538$ for the head alone.
• What retention does the safe LR $2\times 10^{-5}$ give in the toy model? ::: $e^{-0.1}\approx 0.905$, about $90\%$.