Exercises — Transfer learning and fine-tuning

This page is a self-test ladder for the parent topic. Work each problem before opening its solution. The rungs climb from L1 (just recognise the words) to L5 (design and defend a full pipeline).

Every symbol used below is defined the first time it appears. If you have not met a term, the linked prerequisite note builds it: 3.4.1-Introduction-to-CNs, 3.2.3-Gradient-descent-and-backpropagation, 3.2.5-Regularization-techniques, 3.4.7-Batch-normalization, 3.4.8-Residual-connections, 4.1.3-Data-augmentation.

---

Level 1 — Recognition

Goal: can you name the pieces and match them to definitions?

Exercise 1.1

A network is written as $f_{\theta }(x)=g_{{\theta }_{\text{head}}}(h_{{\theta }_{\text{base}}}(x))$.

Here $x$ is one input image, $\theta$ is the full bag of numbers the network can adjust (its weights), and $f_{\theta }(x)$ is the network's output. In plain words, what is $h_{{\theta }_{\text{base}}}$ and what is $g_{{\theta }_{\text{head}}}$? Which part is usually copied from a model trained on a big dataset?

Recall Solution

• $h_{{\theta }_{\text{base}}}$ is the feature extractor (the "base"): the stack of convolutional layers that turns raw pixels into a compact description — think "there is a vertical edge here, fur texture there."
• $g_{{\theta }_{\text{head}}}$ is the head: the small set of final layers that maps that description to the answer we actually want (e.g. "this is species #47").
• The base $h_{{\theta }_{\text{base}}}$ is copied (pretrained) from the big-dataset model. The head is new, because our labels are new.

Exercise 1.2

Match each of the three strategies to its one-line description. (Recall a "block" is one named convolutional stage such as conv5_x, defined at the top of the page.)

Strategies: (A) Feature extraction, (B) Fine-tuning, (C) Full fine-tuning.

Descriptions:

1. Unfreeze all layers; train everything end-to-end with small, layer-wise learning rates.
2. Freeze the whole base; train only the new head.
3. Unfreeze the last few blocks; train them plus the head with a reduced learning rate.

Recall Solution

• A → 2 (freeze base, train only head).
• B → 3 (unfreeze last few blocks).
• C → 1 (unfreeze everything).

The word "freeze" means: set trainable = False, so gradient descent never updates those weights.

Exercise 1.3

True or false, with a one-sentence reason:

"Early convolutional layers (near the input) are the most task-specific and the least transferable."

Recall Solution

False. Early layers learn generic low-level features — edges, colours, simple textures — which appear in almost every visual task, so they are the most transferable. It is the late layers (near the output) that specialise to the source task's exact classes.

---

Level 2 — Application

Goal: plug numbers into the formulas from the parent note.

Exercise 2.1

The parent note states the heuristic $\text{Sample complexity}\propto \frac{\#\text{trainable parameters}}{\text{dataset size}}.$ The symbol $\propto$ means "grows in proportion to." Read it as: if two setups have the same dataset size, the one with more trainable parameters needs proportionally more data to avoid overfitting. It is a rule of thumb, not an equation: there is a hidden constant $k$ so that (roughly) $\text{data-needed}\approx k\cdot \frac{\#\text{trainable}}{\text{dataset size}}$, but $k$ depends on the architecture, task difficulty, and how much regularization you add, so nobody memorises a number for it. You use $\propto$ comparatively: to say "setup X is $47\times$ worse than setup Y," the unknown $k$ cancels and only the ratio of the two right-hand sides matters. "Sample complexity" here is a rough proxy for how much data you need per trainable parameter before the model stops overfitting — bigger ratio means more overfitting risk.

A ResNet-style model has $23,500,000$ total parameters. You freeze the base and keep only a head of $500,000$ trainable parameters. Your dataset has $10,000$ images.

Compute the ratio $\frac{\#\text{trainable}}{\text{dataset size}}$ (a) if you trained all parameters, and (b) with only the head trainable. By what factor did freezing shrink the ratio?

Recall Solution

(a) All trainable: $\frac{23,500,000}{10,000}=2350$.

(b) Head only: $\frac{500,000}{10,000}=50$.

Factor of improvement: $\frac{2350}{50}=47$.

Notice the hidden constant $k$ never appeared: because we took a ratio of two sample-complexity estimates, $k$ cancels. That is the only rigorous way to use a $\propto$ relation quantitatively — compare, never plug in an absolute number. Freezing the base cut the overfitting-risk ratio by $47\times$, so the frozen base acts as strong regularization.

Exercise 2.2

Global Average Pooling. The base outputs a block of numbers of shape $(7,7,2048)$ — read this as "$2048$ channels, each channel being a $7\times 7$ grid." GAP collapses each channel to a single number by averaging its grid: $\text{GAP}(F{)}_c=\frac{1}{7\times 7}{\sum }_{i=1}^7{\sum }_{j=1}^7{F}_{i,j,c}.$

(a) What is the output length (how many numbers) after GAP? (b) For channel $c=1$, suppose its $7\times 7$ grid contains the value $2$ in exactly $12$ cells and $0$ everywhere else. What is $\text{GAP}(F{)}_1$?

Recall Solution

(a) One number per channel, so the output has $2048$ numbers. The spatial $7\times 7$ is averaged away.

(b) The grid has $7\times 7=49$ cells; $12$ hold the value $2$ and the other $37$ hold $0$. Sum $=12\times 2+37\times 0=24$. Divide by $49$: $\text{GAP}(F{)}_1=\frac{24}{49}\approx \mathrm{0.490.}$

Exercise 2.3

Fine-tuning learning rate. The parent rule is ${\text{lr}}_{\text{fine-tune}}=\frac{{\text{lr}}_{\text{initial}}}{10}\text{ to }\frac{{\text{lr}}_{\text{initial}}}{100}.$ Here the learning rate $\eta$ (eta) is the step size in gradient descent: bigger $\eta$ means bolder steps.

You trained the head with ${\text{lr}}_{\text{initial}}=1\times 10^{-3}$. Give the recommended range for the fine-tuning learning rate.

Recall Solution

• Divide by $10$: $1\times 10^{-3}/10=1\times 10^{-4}$.
• Divide by $100$: $1\times 10^{-3}/100=1\times 10^{-5}$.

So ${\text{lr}}_{\text{fine-tune}}\in [1\times 10^{-5},1\times 10^{-4}]$. A common single choice is $1\times 10^{-4}$.

---

Level 3 — Analysis

Goal: reason about why a design choice helps, using the geometry of the loss surface.

Exercise 3.1

Differential (layer-wise) learning rates use ${\eta }_1\text{ (early)},{\eta }_2=3{\eta }_1\text{ (middle)},{\eta }_3=10{\eta }_1\text{ (head)}.$

(a) If ${\eta }_1=1\times 10^{-5}$, compute ${\eta }_2$ and ${\eta }_3$. (b) In one sentence each, explain why early layers get the smallest rate and the head the largest.

Figure — reading the schematic below. The picture is a deliberately simplified 1-D cross-section of the loss landscape, drawn to contrast step sizes, not to depict a real network. Read every element as follows:

• Horizontal axis = the value of one representative weight (imagine collapsing all the millions of weights onto a single left–right dial).
• Vertical axis = the loss $\mathcal{L}$, i.e. how wrong the model currently is; lower is better.
• The lavender curve is a "valley": the lowest point is the good minimum, marked by the butter dot labelled good minimum.
• The short mint arrow on the left is one gradient-descent step for the early layers: its small length shows the small step size ${\eta }_1$ — barely a nudge, so the good features are preserved.
• The long coral arrow on the right is one step for the head: its large length shows the large step size ${\eta }_3$ — a bold move, because the head starts random and must travel far.
• The length of each arrow equals its learning rate (step size); direction is "downhill toward lower loss." Real loss surfaces live in millions of dimensions — this single slice merely lets us see "small step vs. big step."

Recall Solution

(a) ${\eta }_2=3\times (1\times 10^{-5})=3\times 10^{-5}$. ${\eta }_3=10\times (1\times 10^{-5})=1\times 10^{-4}$.

(b)

• Early layers: they already encode near-universal features (edges/colours) that are correct for almost any vision task — so we want tiny nudges only. Look at the short mint arrow on the left of the figure.
• Head: it started from random weights and knows nothing about our labels, so it needs big steps to reach a good state fast — the long coral arrow on the right.

Exercise 3.2

Catastrophic forgetting as a distance. The parent note wants the base to stay close to its pretrained point: $\Vert {\theta }_{\text{base}}^{t+1}-{\theta }_{\text{base}}^{\text{pretrain}}\Vert \ll 1.$ The symbol $\Vert \cdot \Vert$ is the ==Euclidean (${\ell }_2$) norm==: for a vector $v=(v_1,v_2,\dots ,v_n)$ it is $\Vert v\Vert =\sqrt{v_1^2+v_2^2+\cdots +v_n^2},$ the ordinary straight-line length you get from the Pythagorean theorem, extended to $n$ dimensions. So $\Vert {\theta }_{\text{base}}^{t+1}-{\theta }_{\text{base}}^{\text{pretrain}}\Vert$ is the straight-line distance between the new weight vector and the pretrained weight vector — literally "how far, in total, did all the weights move together." We want it $\ll 1$ (much less than one) so the base barely budges.

Suppose after one gradient step a single weight moved from ${\theta }^{\text{pretrain}}=0.400$ to ${\theta }^{t+1}=0.406$ using $\eta =1\times 10^{-4}$.

(a) What was the size of the gradient $\nabla \mathcal{L}$ for that weight? (Use ${\theta }^{t+1}={\theta }^t-\eta \nabla \mathcal{L}$ with ${\theta }^t=0.400$.) (b) If instead $\eta =1\times 10^{-3}$ (ten times bigger) with the same gradient, where would the weight land, and how far did it move?

Recall Solution

(a) Rearrange: $\nabla \mathcal{L}=\frac{{\theta }^t-{\theta }^{t+1}}{\eta }=\frac{0.400-0.406}{1\times 10^{-4}}=\frac{-0.006}{1\times 10^{-4}}=-60$.

(b) New position: ${\theta }^{t+1}=0.400-(1\times 10^{-3})(-60)=0.400+0.060=0.460$. It moved $0.060$ — ten times farther than the $0.006$ move in (a).

For a single weight the ${\ell }_2$ norm of the change is just $|0.006|=0.006$ vs. $|0.060|=0.060$; the small-$\eta$ case keeps that distance far below $1$, the large-$\eta$ case starts drifting. This makes the danger concrete: same gradient, $10\times$ the learning rate, $10\times$ the drift away from the good pretrained value. Keep $\eta$ small to keep the drift small.

---

Level 4 — Synthesis

Goal: combine multiple ideas into a coherent design.

Exercise 4.1

You have three projects. For each, pick (A) feature extraction, (B) partial fine-tuning, or (C) full fine-tuning, and justify in two sentences. (Recall "block" = one named convolutional stage, e.g. conv5_x, per the definition at the top.)

1. $600$ labelled photos of $3$ dog breeds. Source: ImageNet (which contains many dog breeds).
2. $40,000$ labelled retinal (eye) scans, $5$ disease classes. Source: ImageNet natural photos.
3. $150,000$ labelled satellite tiles, $12$ land-use classes. Source: ImageNet.

Recall Solution

1. A — feature extraction. Tiny dataset ($<1000$) and very similar to the source (dogs are in ImageNet). The frozen base already extracts everything relevant; only a head is needed. Unfreezing would overfit $600$ images fast.
2. B — partial fine-tuning. Medium dataset (1k–100k) but a domain shift (medical vs. natural). Keep the early edge/texture stages frozen (still useful, since retinal images still contain edges and textures) and unfreeze only the last block(s) — e.g. conv5_x — so the highest-level features can re-specialise to the appearance of retinas.
3. C — full fine-tuning. Large dataset ($>100$k) and a big domain gap (top-down satellite vs. natural photos). Enough data to safely move all layers with differential, small learning rates for maximum performance.

Exercise 4.2

Order the pipeline. Arrange these five steps into the correct sequence and give the one-line reason for the first step's placement. ("Last block(s)" here means the final one or two named stages such as conv5_x.)

• (i) Reduce learning rate by $10\times$.
• (ii) Train the head with the base frozen for a few epochs.
• (iii) Load pretrained base, freeze it, attach a fresh head.
• (iv) Unfreeze the last block(s).
• (v) Continue training base + head jointly.

Recall Solution

Correct order: (iii) → (ii) → (iv) → (i) → (v).

• (iii) build the model with frozen base + new head;
• (ii) warm up the head so it is no longer random;
• (iv) unfreeze the last block(s);
• (i) lower the learning rate;
• (v) train jointly.

Why (ii) — head warm-up — must come before unfreezing: a freshly-initialised head emits large, meaningless gradients. If the base is unfrozen while the head is still random, those junk gradients flow back and corrupt the pretrained features (catastrophic forgetting) before the head ever becomes useful.

---

Level 5 — Mastery

Goal: quantify a full pipeline end-to-end and defend it with numbers.

Exercise 5.1 — Bird species pipeline

Recreate the parent note's example. Source: ResNet-style base on ImageNet. Target: $200$ bird species, $50$ images each.

(a) Total target images? (b) The base outputs shape $(7,7,2048)$; after GAP and a $\text{Dense}(512)$ layer, how many weights connect GAP-output to that dense layer? (Weights only, ignore biases: it is $2048\times 512$.) (c) The final classification layer is $\text{Dense}(200)$ fed by the $512$ features. How many weights there (again $\text{inputs}\times \text{outputs}$)? (d) Feature-extraction gave $\approx 75\%$ validation accuracy; fine-tuning the last block raised it to $\approx 87\%$. State the absolute improvement in percentage points.

Recall Solution

(a) $200\times 50=10,000$ images — a medium dataset, so fine-tuning is appropriate.

(b) GAP produces $2048$ numbers; connecting to $512$ units: $2048\times 512=1,048,576$ weights.

(c) $512\times 200=102,400$ weights.

(d) $87\%-75\%=12$ percentage points. The gain comes from the last block (conv5_x) re-learning high-level bird-part features that generic ImageNet layers did not specialise for.

Exercise 5.2 — Effective regularization check

Continuing 5.1: the new head's trainable weights are $1,048,576$ (dense-512) $+102,400$ (dense-200) $=1,150,976$, plus a small BatchNorm and bias count we round to a total head of $\approx 1,150,000$ trainable parameters. Total model parameters $\approx 23,500,000$.

(a) In feature-extraction mode, what fraction of parameters is trainable (as a percentage)? (b) The parent note says heads are typically $0.1\%$–$10\%$ of the total. Does this fall in range?

Recall Solution

(a) $\frac{1,150,000}{23,500,000}=0.04894\dots \approx 4.89\%$.

(b) Yes — $4.89\%$ sits comfortably inside the $0.1\%$–$10\%$ band. Only about $1$ in $20$ parameters is learnable, which is exactly the strong-regularization effect that lets $10,000$ images train the model without severe overfitting.

---

Recall Quick self-check reveals

What does freezing the base do to the trainable-parameter ratio, and why does that help small datasets? ::: It shrinks the ratio dramatically (e.g. $47\times$ in Ex 2.1), acting as strong regularization so the model cannot overfit a tiny dataset. Why warm up the head before unfreezing the base? ::: A random head emits large junk gradients that would corrupt pretrained features (catastrophic forgetting) if the base were already unfrozen. Why use layer-wise learning rates with ${\eta }_{\text{early}}<{\eta }_{\text{head}}$? ::: Early layers hold near-universal features needing only tiny nudges; the head starts random and needs bold steps. What is a "block" in a ResNet-style network? ::: One of the four named residual convolutional stages (conv2_x–conv5_x) that the network treats as a single freeze/unfreeze unit.