Foundations — Transfer learning and fine-tuning

Before you can read the parent note Transfer learning and fine-tuning, every squiggle in its equations must mean something concrete to you. This page picks up every symbol and word the parent uses and builds it from nothing. Read top to bottom — each idea leans on the one above it.

1. What is a "model"? The function $f$

We write one use of the machine as $f(x)$, read aloud as "$f$ of $x$": "the answer the machine gives when I feed it $x$."

• $x$ (chalk-blue in the figure) = the input, here an image (a grid of pixel numbers).
• $f(x)$ = the output, here a guess like "this is a robin."

Look at the figure: $x$ goes in the left, travels through the box $f$, and a label pops out the right. That box is the whole reason we build networks. If you have met the pixel-grid idea in 3.4.1-Introduction-to-CNs, this is the same picture.

2. The tuning knobs: parameters $\theta$

The box $f$ has thousands of little dials inside it. Turning the dials changes what guess comes out. We bundle all those dial settings into one symbol: $\theta$ (the Greek letter theta, said "THAY-ta").

Whenever the parent writes ${\theta }_{\text{base}}$ or ${\theta }_{\text{head}}$, it just means "the dials belonging to this particular part of the machine." We build those parts in §7.

3. Data, the pair $(x,y)$, and the dataset $\mathcal{D}$

To learn, the machine needs examples where we already know the right answer.

A whole pile of these flashcards is the dataset, written $\mathcal{D}$ (a fancy curly $D$).

The parent uses two piles:

• ${\mathcal{D}}_{\text{source}}$ — the huge pile the model first learned on (ImageNet, ~14 million pictures).
• ${\mathcal{D}}_{\text{target}}$ — your small pile (the new task, maybe a few thousand).

The entire point of transfer learning is: knowledge squeezed out of ${\mathcal{D}}_{\text{source}}$ helps us when ${\mathcal{D}}_{\text{target}}$ is too small to learn from alone.

4. Being wrong on purpose: the loss $\mathcal{L}$

How do we know if a dial setting is good? We measure how wrong the guess is.

The parent's loss is categorical_crossentropy. You do not need its formula here — just hold the picture: a scoreboard that punishes wrong guesses. Lower is better.

5. Averaging over the pile: $\mathbb{E}$

One flashcard's score isn't enough — we care about doing well on average across the whole pile.

Put §4 and §5 together and the parent's core objective reads in plain English:

${\theta }^{\ast }=\arg {\min }_{\theta }{\mathbb{E}}_{(x,y)\sim \mathcal{D}}[\mathcal{L}(f_{\theta }(x),y)]$

6. Layers, and the "early → late" ladder

A CN is not one box but a stack of boxes called layers. Each layer transforms the picture a little and passes it to the next.

The figure shows the ladder the parent leans on constantly:

• Early layers (near the input) → generic edges, colours, simple textures. These look the same for cats, X-rays, or satellites — that's why they transfer.
• Middle layers → object parts (an eye, a wheel).
• Late layers (near the output) → the actual answer classes, tightly tied to the source task.

This ladder is the whole reason freezing works: the bottom rungs are reusable, only the top rungs are task-specific. If layers and convolutions are new, pause and read 3.4.1-Introduction-to-CNs.

7. Splitting the machine: base $h$ and head $g$

Transfer learning cuts the stack into two pieces.

The parent writes the reassembled machine as:

$f_{\theta }(x)=g_{{\theta }_{\text{head}}}(h_{{\theta }_{\text{base}}}(x))$

The lock icon on the base means frozen — its dials stay fixed. The green head is what we actually train. This single picture is the heart of the whole parent note.

8. Freezing, and the sample-complexity ratio

Why freeze? The parent gives a ratio:

$\text{Sample complexity}\propto \frac{\text{\# trainable parameters}}{\text{dataset size}}$

9. Nudging the dials: the update rule and $\eta$

Training turns the dials in small steps. One step looks like:

${\theta }^{t+1}={\theta }^t-\eta {\nabla }_{\theta }\mathcal{L}$

Let's decode every piece.

The parent even uses different step sizes per part: tiny ${\eta }_1$ for early layers (keep the general edges), a larger ${\eta }_3$ for the head (it needs to learn a lot). Same idea, applied layer by layer.

10. Squeezing the feature map: Global Average Pooling

The base $h$ hands back not a flat list but a little stack of grids of shape $(7,7,2048)$.

To feed the head, we flatten each $7\times 7$ map to a single number by averaging it:

$\text{GAP}(F{)}_c=\frac{1}{7\times 7}{\sum }_{i=1}^7{\sum }_{j=1}^7{F}_{i,j,c}$

You'll also see 3.4.7-Batch-normalization and 3.4.8-Residual-connections named in the head and inside ResNet50; those are covered in their own notes — here just know they're standard layers you can drop in.

Prerequisite map

Every arrow is a "you need this before that." The bottom node is exactly the parent topic.

Equipment checklist

Cover the right side and answer each. If you can, you're ready for the parent note.