Worked examples — Transfer learning and fine-tuning

Before any example, let's re-anchor the symbols we'll use constantly, in plain words:

The scenario matrix

Every transfer-learning decision is really an answer to two questions: how much target data do I have? and how similar is my task to the source task? Plus a few edge situations. Here is the full grid — every example below is tagged with the cell it fills.

Cell	Data size	Task similarity	Special condition	Right move
A	tiny (<1k)	high	—	Feature extraction (freeze all)
B	tiny (<1k)	low	—	Freeze all, but expect trouble
C	medium (1k–100k)	high	—	Fine-tune last block
D	medium	low	—	Fine-tune more blocks
E	large (>100k)	any	—	Full fine-tuning
F	any	any	learning rate too big	Catastrophic forgetting (degenerate)
G	any	any	head not warmed up	Random gradients corrupt base (degenerate)
H	zero-ish (per-class ~5)	high	limiting case	Extreme few-shot; linear head only
I	real-world word problem	—	budget/latency limit	Trade-off decision
J	exam twist	—	differential LR arithmetic	Compute layer-wise steps

We'll now cover all ten cells with worked examples.

---

Example 1 — Cell A: tiny data, high similarity

Forecast: Before reading on — freeze or fine-tune? How many trainable params: thousands, or millions?

1. Decide strategy. 800 samples is tiny and the task is similar → feature extraction, freeze the whole base. Why this step? With so few images, training millions of weights would overfit instantly — the sample-complexity ratio $R=\frac{\#\text{trainable}}{\text{data size}}$ must stay small.
2. Count trainable parameters. The frozen base outputs a $2048$-length vector after global average pooling. We add one dense layer to 4 classes: $\#{\theta }_{\text{head}}=\underset{\text{weights}}{\underbrace{2048\times 4}}+\underset{\text{biases}}{\underbrace{4}}=8196$ Why this step? This is the entire trainable count. Everything else is frozen.
3. Ratio check. $R=\frac{8196}{800}\approx 10.2$ Why this step? Compare against training all $25,600,000$ ResNet50 params: that ratio would be $32,000$ — hopeless.

See the frozen-vs-trainable split below (red = the only part that learns).

---

Example 2 — Cell C: medium data, high similarity, fine-tune last block

Forecast: Guess the fine-tune rate and the percentage-point jump.

1. Apply the rule. From the parent: ${\eta }_{\text{fine-tune}}={\eta }_{\text{initial}}/10$ to ${\eta }_{\text{initial}}/100$. ${\eta }_{\text{fine-tune}}\in \left[\frac{10^{-3}}{100},\frac{10^{-3}}{10}\right]=[10^{-5},10^{-4}]$ Why this step? Small steps keep ${\theta }_{\text{base}}$ inside the basin of the pretrained minimum, so we adapt without erasing edge/texture detectors.
2. Pick $\eta =10^{-4}$ (the aggressive end, safe because task is similar).
3. Compute the gain. $\Delta \text{acc}=87\%-75\%=12\text{percentage points}$ Why this step? Block 5 held ImageNet-specific "this is a mammal" concepts; retraining it lets it hold "this is a warbler-vs-finch" concepts instead.

---

Example 3 — Cell J (exam twist): differential learning rates

Forecast: The head should move much more than the early layers — but exactly how much more?

1. Compute each rate. ${\eta }_1=10^{-5},{\eta }_2=3\times 10^{-5},{\eta }_3=10\times 10^{-5}=10^{-4}$ Why this step? Early layers hold universal edges → nudge them least; head is task-specific → nudge it most.
2. Compute each step with $\nabla =2$: $\Delta {\theta }_{\text{early}}=2\times 10^{-5}=2\times 10^{-5}$ $\Delta {\theta }_{\text{mid}}=2\times 3\times 10^{-5}=6\times 10^{-5}$ $\Delta {\theta }_{\text{head}}=2\times 10^{-4}=2\times 10^{-4}$
3. Ratio. $\frac{\Delta {\theta }_{\text{head}}}{\Delta {\theta }_{\text{early}}}=\frac{2\times 10^{-4}}{2\times 10^{-5}}=10$ Why this step? Confirms the design intent: the head adapts $10\times$ faster than the base.

The step sizes are drawn to scale below (red = head step, the tallest).

---

Example 4 — Cell F (degenerate): catastrophic forgetting from a big rate

Forecast: Is the moved distance "tiny" ($\ll 1$) as fine-tuning requires, or huge?

1. Update with the huge rate. ${\theta }^{t+1}={\theta }^{\text{pre}}-\eta \nabla =0.40-(10^{-2})(3)=0.40-0.03=0.37$ Distance moved in one step $=0.03$.
2. Now over many steps / many weights the moves accumulate. Contrast the safe rate $\eta =10^{-4}$: $0.40-(10^{-4})(3)=0.39970,\text{move}=3\times 10^{-4}$ Why this step? The safe step is $100\times$ smaller — that is the whole point of dividing the rate.
3. Interpret. The condition is $\Vert {\theta }^{t+1}-{\theta }^{\text{pre}}\Vert \ll 1$ — recall this is the straight-line (Euclidean) distance of the whole weight-point from the pretrained point. A single $0.03$ move looks fine, but with $10^{-2}$ it compounds across 25M weights and many epochs, so the point drifts far, leaving the basin and overwriting useful edge/texture detectors — catastrophic forgetting.

The two trajectories away from the pretrained minimum are sketched below (red = the reckless big-rate jump out of the basin).

---

Example 5 — Cell G (degenerate): head not warmed up

Forecast: Will freezing-first or unfreezing-first give a bigger, more damaging base update on epoch 1?

1. Immediate-unfreeze base step. The huge head gradient back-propagates into the base: $\Delta {\theta }_{\text{base}}=\eta \nabla =10^{-4}\times 8=8\times 10^{-4}$ per weight, per step, driven by noise.
2. After warm-up. Once the head is trained (loss small), gradients shrink to $\nabla \approx 0.5$: $\Delta {\theta }_{\text{base}}=10^{-4}\times 0.5=5\times 10^{-5}$ Why this step? $16\times$ smaller — now the base only feels meaningful signal.
3. The fix. Train head-with-frozen-base first (feature-extraction phase), then unfreeze.

---

Example 6 — Cell E: large data, full fine-tuning is safe

Forecast: With 500k images, is training 25.6M weights over- or under-constrained?

1. Compute the ratio $R=\frac{\#\text{trainable}}{\text{data size}}$: $R=\frac{2.56\times 10^7}{5\times 10^5}=51.2$ Why this step? Compare to Example 1's frozen case ($R\approx 10$) — same order of magnitude, so full training is now affordable, unlike the 800-image case where it was $32,000$.
2. Justify unfreezing all. Task is very different (satellite ≠ natural photos), so even early edge filters may need retuning → freezing them would waste capacity.
3. Still use small $\eta$ + differential rates to start from the pretrained point rather than random. Why this step? Even when every layer is trainable, starting from the pretrained weight-point places us in an already-good region of weight space (near a useful basin), so we converge in far fewer epochs than random initialization — and small $\eta$ keeps early edge filters from being trashed before they've proven unhelpful.

---

Example 7 — Cell H (limiting case): 5 images per class, few-shot

Forecast: Which head can the 15 samples support without total overfit — hidden or linear?

1. Linear head params: $2048\times 3+3=6147$.
2. 256-hidden head params: $(2048\times 256+256)+(256\times 3+3)=524544+771=525315$. Why this step? The hidden layer explodes the count $\sim 85\times$.
3. Ratios against 15 samples. $R_{\text{linear}}=\frac{6147}{15}\approx 410,R_{\text{hidden}}=\frac{525315}{15}\approx 35021$ Why this step? Both are large, but the hidden head is $85\times$ worse — in the few-shot limit you must use the linear head only (and strong dropout / augmentation from 4.1.3-Data-augmentation).

---

Example 8 — Cell I (real-world word problem): budget & latency

Forecast: Is a 2-point accuracy gain worth breaking the latency budget?

1. Latency constraint. Full fine-tune leaves a heavier, unquantized model at ~40 ms > 30 ms budget → fails the hard constraint. Why this step? A hard constraint (real-time) beats a soft metric (2% accuracy).
2. Data size. 3,000 with high similarity (food is common in ImageNet) → feature extraction already near-optimal.
3. Decision. Ship the frozen-base, quantized model at 82% / 22 ms. Why this step? $22<30$ ms passes; the extra 2% cannot justify violating latency.

---

Example 9 — Cell D: medium data, low similarity

Forecast: Similar task → few blocks. Different task → ?

1. Similarity logic. Ultrasound ≠ ImageNet, so even mid-level textures (held in blocks 3–4) differ → those mid stages need adapting too. Why this step? Only the earliest edge filters (blocks 1–2) stay universal; the rest must move.
2. Ratio at blocks 3–5. $R=\frac{2.1\times 10^7}{2\times 10^4}=1050$ Why this step? Large but tolerable with augmentation + 3.2.5-Regularization-techniques; the +6-point gain (79→85) confirms it paid off.
3. Contrast with birds (Ex. 2): high similarity → block 5 alone sufficed for +12. Low similarity → need blocks 3–5 for +6.

---

Example 10 — Cell B: tiny data, low similarity (the hard corner)

Forecast: With only 600 foreign images, can you fine-tune at all — and will frozen ImageNet features even help?

1. Ratio if you tried fine-tuning block 5 ($\approx 1.5\times 10^7$ trainable params): $R=\frac{1.5\times 10^7}{600}=25000$ Why this step? This $R$ is astronomically over-parameterized (compare Example 1's safe $R\approx 10$) → guaranteed overfit → fine-tuning is forbidden. You are forced to freeze the whole base and train only a small head.
2. But frozen features are a poor match. Thermal images have no natural RGB edges/textures, so ImageNet's block-1 filters fire weakly → expect only mediocre accuracy (say ~60%), well below the similar-task cases. Why this step? This is the genuinely hard cell: too little data to adapt the base, yet the base's features are too foreign to reuse well — both levers are jammed.
3. Mitigation. Lean hard on 4.1.3-Data-augmentation to stretch the 600 images, use strong dropout + 3.2.5-Regularization-techniques on the tiny head, and if at all possible pretrain on a closer source domain (e.g. a grayscale or thermal dataset) or collect more data — the only real cures for Cell B.

---

Recall

Recall Which cell forces a linear head only?

Cell H (few-shot, ~5 images/class) — a hidden layer's ~525k params against 15 samples overfits catastrophically. ::: Linear head, $\approx 6147$ params.

Recall Big learning rate on an unfrozen base causes what?

Catastrophic forgetting — weights leave the pretrained basin and useful features are overwritten (Example 4). ::: Drop $\eta$ by $10$–$100\times$.

Recall Why warm up the head before unfreezing the base?

A random head emits large noisy gradients that back-propagate and corrupt the base (Cell G). ::: Freeze → train head → then unfreeze.

Recall What does the sample-complexity ratio

$R$ measure, and what makes it small? $R=\frac{\#\text{trainable parameters}}{\text{dataset size}}$; freezing layers shrinks the numerator so $R$ stays small and the model generalizes. ::: See 3.2.3-Gradient-descent-and-backpropagation.

Recall Which ResNet50 block is nearest the output, and what does it hold?

Block 5 (conv5_x), the last stage — high-level, task-specific object concepts. ::: Unfrozen first during fine-tuning.

Related building blocks: 3.4.1-Introduction-to-CNs, 3.4.7-Batch-normalization, 3.4.8-Residual-connections.