Visual walkthrough — Transfer learning and fine-tuning

Everything below rests on one mental image: learning = walking downhill on a landscape of error. We will earn that image first, then let the rest fall out of it.

---

Step 1 — What "training a network" even means, as a picture

WHAT. A neural network is just a big box of adjustable numbers. Call the whole collection of those numbers $\theta$ (the Greek letter theta — read it as "the settings"). Feeding a picture $x$ into the box with settings $\theta$ produces a guess, written $f_{\theta }(x)$. We compare the guess to the true answer $y$ and measure how wrong we are with a loss $\mathcal{L}$ — a single positive number, bigger = more wrong.

WHY a landscape? Because $\mathcal{L}$ depends on the settings $\theta$, we can imagine plotting wrongness (height) versus settings (position on the ground). That surface is the loss landscape. Training means: find the lowest valley.

$${\theta }^{\ast }=\underset{\genfrac{}{}{0ex}{}{\text{"which settings}}{\text{make loss smallest?"}}}{\underbrace{\arg \underset{\theta }{\min }}}\underset{\text{how wrong the guess is}}{\underbrace{\mathcal{L}(f_{\theta }(x),y)}}$$

• ${\theta }^{\ast }$ — the best settings, the star marks "answer we want."
• $\arg \min$ — literally "the argument (the $\theta$) that produces the minimum," i.e. where the valley bottom is, not how deep it is.

PICTURE. A green ball sits somewhere on a curvy surface. The height under the ball is the loss.

---

Step 2 — Why a small dataset makes the valley untrustworthy

WHAT. The landscape isn't fixed — its exact shape is drawn by your data. Every training example carves the surface a little. With millions of examples the surface is smooth and honest: its true valley is the true best network. With only a handful, the surface is jagged and overfit — full of fake potholes that fit your few points but nothing else.

WHY this matters. The parent note's ratio,

$$\text{sample complexity}\propto \frac{\#\text{trainable parameters}}{\text{dataset size}},$$

• numerator $\#\text{trainable parameters}$ — how many dials the ball can wiggle = how wrinkly the landscape can be,
• denominator $\text{dataset size}$ — how many examples smooth those wrinkles out,

says: many dials + few examples = a landscape of fake potholes. The ball will happily fall into one and think it won.

PICTURE. Left: smooth valley from lots of data, honest bottom. Right: same region drawn from few points — spiky, with a deceptive fake minimum the ball gets stuck in.

---

Step 3 — The pretrained model hands you a starting position, not a blank one

WHAT. We split the network into two pieces:

$$f_{\theta }(x)=\underset{\text{the "decider"}}{\underbrace{g_{{\theta }_{\text{head}}}}}\left(\underset{\text{the "feature reader"}}{\underbrace{h_{{\theta }_{\text{base}}}(x)}}\right)$$

• $h_{{\theta }_{\text{base}}}$ — the base: the tall stack of convolutional layers that turns raw pixels into meaningful features (edges → textures → parts). Its settings ${\theta }_{\text{base}}$ come pretrained from ImageNet.
• $g_{{\theta }_{\text{head}}}$ — the head: a small final layer that maps those features to your labels. Its settings ${\theta }_{\text{head}}$ start random.

WHY it helps. Standard training drops the ball at a random spot — likely on a far mountainside. Pretraining drops the base part of the ball already deep in a good valley that a million ImageNet pictures dug. From there, only a short walk remains. (Those "features" are exactly the edge/texture hierarchy from 3.4.1-Introduction-to-CNs.)

PICTURE. Two balls on the same landscape: red starts on a distant peak (random init), green starts already inside a valley (pretrained init). The green ball's remaining journey is tiny.

---

Step 4 — Strategy 1 (Feature Extraction): freeze the ball's base, move only the head

WHAT. Set ${\theta }_{\text{base}}$ frozen (unmovable). Only the head ${\theta }_{\text{head}}$ can change. The head learns a simple map from the already-good features to your labels:

$$\hat{y}=\text{softmax}(\underset{\text{head weights}}{\underbrace{W}}\underset{\text{frozen features}}{\underbrace{h_{{\theta }_{\text{base}}}(x)}}+\underset{\text{bias}}{\underbrace{b}})$$

• $W,b$ — the only movable dials now.
• $\text{softmax}$ — squashes a list of scores into probabilities that sum to $1$ (so "how bird-like" becomes "$83\%$ robin, $9\%$ sparrow, ...").

WHY it's safe. Freezing the base collapses the number of movable dials to almost nothing. Back to Step 2's ratio: numerator shrinks by ~$99\%$, so the effective landscape becomes smooth and honest even on tiny data. Freezing is regularization.

PICTURE. The base of the ball is bolted to the ground (frozen); only a small top slider moves along a smooth low-dimensional groove to its minimum.

Recall

Why does freezing the base help on a small dataset? ::: It removes almost all trainable dials, so the loss landscape can't grow fake potholes — the classic bias/variance trade of 3.2.5-Regularization-techniques.

---

Step 5 — Strategy 2 (Fine-tuning): why we unfreeze, and why gently

WHAT. Feature-extraction alone assumes ImageNet's features are already perfect for birds. Usually the last layers (which encode specific concepts) could adapt a little. So we unfreeze the last few blocks and let ${\theta }_{\text{base}}$ move — but with a tiny step size $\eta$ (eta = the learning rate, how far the ball jumps each step):

$${\theta }_{\text{base}}^{t+1}={\theta }_{\text{base}}^t-\underset{\text{step size}}{\underbrace{\eta }}\underset{\text{downhill direction}}{\underbrace{{\nabla }_{{\theta }_{\text{base}}}{\mathcal{L}}_{\text{target}}}}$$

• $\nabla \mathcal{L}$ — the gradient, an arrow pointing in the steepest-uphill direction; the minus sign flips it to downhill.
• $\eta$ — how big a leap we take along that arrow.

WHY small $\eta$. The pretrained base already sits in a good valley. A big $\eta$ leaps clear across it and lands back on a mountainside — the network forgets its ImageNet skill. That is catastrophic forgetting. A small $\eta$ keeps

$$\Vert {\theta }_{\text{base}}^{t+1}-{\theta }_{\text{base}}^{\text{pretrain}}\Vert \ll 1,$$

• $\Vert \cdot \Vert$ — length of the change vector = "how far we drifted from the pretrained spot."

so we only nudge within the good basin.

PICTURE. Same valley, two ball trajectories: a huge yellow jump overshoots out of the basin (forgetting); a chain of tiny green steps settles gently into the true bottom.

---

Step 6 — Differential learning rates: slower for the base, faster for the head

WHAT. Not all layers should move equally. Early layers (edges) are universal → barely touch them. Later layers and the head are task-specific → let them move more. We give each depth its own step size:

$$\begin{array}{rlrl}{\theta }_{\text{early}}^{t+1}& ={\theta }_{\text{early}}^t-{\eta }_1\nabla \mathcal{L}& & ({\eta }_1\text{ tiny})\\ {\theta }_{\text{middle}}^{t+1}& ={\theta }_{\text{middle}}^t-{\eta }_2\nabla \mathcal{L}& & ({\eta }_2=3{\eta }_1)\\ {\theta }_{\text{head}}^{t+1}& ={\theta }_{\text{head}}^t-{\eta }_3\nabla \mathcal{L}& & ({\eta }_3=10{\eta }_1)\end{array}$$

• ${\eta }_1<{\eta }_2<{\eta }_3$ — step size grows with depth. Bottom of the network moves in millimetres; top moves in metres.

WHY. Edges look identical in birds, MRIs and satellites — no reason to disturb them. The final "which bird?" logic is what's genuinely new. Matching step size to how-much-adapting-is-needed spends your movement budget where it counts. (This layered stability is also why 3.4.7-Batch-normalization and 3.4.8-Residual-connections make fine-tuning behave.)

PICTURE. The network drawn as a vertical stack; short arrows on the bottom layers, medium in the middle, long on top — a staircase of step sizes.

---

Step 7 — Edge cases: when the picture breaks

WHAT & WHY, case by case:

• Target task = source task exactly. The valley is already the right one. Movement budget ≈ 0 → pure feature extraction (Step 4). Unfreezing only risks forgetting for no gain.
• Target task wildly different (birds → seismograms). ImageNet's valley is on the wrong continent of the landscape. A gentle nudge won't reach the new valley → you need full fine-tuning with more movement, or even fresh training. The green ball's head-start is small here.
• Dataset ≈ 0 (a few dozen images). Even the head overfits. Freeze everything, use the frozen features with a simple classifier, and lean hard on data augmentation to fake more examples.
• Dataset huge (millions). The honest smooth landscape returns (Step 2 denominator wins). Pretraining now only saves time, not accuracy — full fine-tuning is fine.

PICTURE. A 2-axis map: horizontal = task similarity, vertical = dataset size. Four quadrants labelled with the strategy that wins in each — the decision the whole page has been building toward.

---

The one-picture summary

The entire derivation is one landscape: random init starts far up a slope over a wrinkled (small-data) surface and gets stuck in a fake pothole; pretrained init starts inside a good valley dug by millions of images; feature extraction bolts the base and slides only the head to the local bottom; fine-tuning takes tiny steps to settle the base into the true bottom — while a reckless big step leaps out of the basin entirely (catastrophic forgetting).

Recall Feynman retelling — say it like a story

You already know how to see. Someone spent a million pictures teaching a machine to see too — to notice edges, then fur, then whole beaks. That skill lives as a position in a giant landscape of "wrongness," parked deep in a good valley. When you show up with only ten thousand bird photos, you don't send the machine back to the top of a random mountain. You let it keep its parking spot. First you bolt its eyes in place and only teach the last little decider "which bird?" — that's feature extraction, and because almost nothing can move, your tiny dataset can't dig fake traps. If birds need slightly sharper eyes, you gently unfreeze the top few layers and nudge — tiny steps, so it never jumps out of its good valley and forgets how to see. You nudge the deep edge-detectors least (edges are edges everywhere) and the top-most decider most (that's the genuinely new part). And if your new task is nothing like pictures, or you have millions of examples, the head-start matters less and you let it walk further. That's the whole thing: keep the good starting spot, move only as much as your data can safely justify.

---

Connections

Parent: Transfer learning and fine-tuning · Hinglish: 3.4.11 Transfer learning and fine-tuning (Hinglish) Prerequisites used above: 3.4.1-Introduction-to-CNs, 3.2.3-Gradient-descent-and-backpropagation, 3.2.5-Regularization-techniques, 3.4.7-Batch-normalization, 3.4.8-Residual-connections, 4.1.3-Data-augmentation.