Visual walkthrough — Full fine-tuning vs feature extraction

We assume you have never seen a neural network diagram before. Every arrow, every symbol, is earned before it is used.

---

Step 1 — What a model actually is: a two-stage machine

WHAT. A trained model is just a machine that turns an input $x$ (a sentence, an image) into a prediction $\hat{y}$ (a label, "positive" or "negative"). But it does this in two stages stacked one after the other.

WHY split it in two? Because the two stages come from two different places. The first stage was learned on enormous general data (pretraining). The second stage is your new part, added for your task. Keeping them separate is the whole story of this page — you get to treat them differently.

PICTURE. Look at the figure. The input $x$ (left) flows through a big grey box, the backbone, then a small box, the head, out to the prediction $\hat{y}$ (right).

Written as one composed machine:

$\underset{\text{prediction}}{\underbrace{\hat{y}}}=\underset{\begin{array}{c}\text{head}\\ \text{(your part)}\end{array}}{\underbrace{h_{\varphi }}}\left(\underset{\begin{array}{c}\text{backbone reads }x\\ \text{→ feature vector}\end{array}}{\underbrace{f_{\theta }(x)}}\right)$

Read it inside-out: first $f_{\theta }(x)$ runs, its output is handed to $h_{\varphi }$. A "knob" is just a number the machine can adjust; $\theta$ is millions of knobs, $\varphi$ is a handful.

---

Step 2 — How a machine learns: nudging knobs downhill

WHAT. Learning = repeatedly measuring how wrong the machine is, then nudging each knob a tiny bit in the direction that reduces the wrongness.

WHY do we need a "direction"? Because there are millions of knobs; we can't try all combinations. We need, for each knob, an arrow saying "turning me this way lowers the error." That arrow is the gradient.

PICTURE. Imagine the error as a hilly landscape and the knob setting as a ball on it. The gradient points uphill; we step the opposite way, downhill, toward lower error. The red ball is the current knob value; the red arrow is the downhill step.

The update rule for any knob $w$:

$\underset{\text{knob}}{\underbrace{w}}\leftarrow \underset{\text{old value}}{\underbrace{w}}-\underset{\begin{array}{c}\text{step size}\\ (\text{"eta"})\end{array}}{\underbrace{\eta }}\underset{\begin{array}{c}\text{downhill}\\ \text{direction}\end{array}}{\underbrace{\frac{\partial \mathcal{L}}{\partial w}}}$

• $\leftarrow$ means "replace the left with the right."
• The minus sign is what makes us go downhill (opposite the uphill gradient).
• $\eta$ ("eta") is the learning rate: how big a step we take. Small $\eta$ = timid steps; large $\eta$ = bold steps. Hold this word — it becomes the star of Step 6.

The one crucial fact: a knob only moves if its gradient is computed and applied. Kill the gradient, and the knob is frozen in place. That single lever is the entire difference between our two strategies.

---

Step 3 — Feature extraction: turn the backbone to stone

WHAT. We declare every backbone knob $\theta$ a constant. It receives no gradient, so it never moves. Only the head knobs $\varphi$ learn.

WHY? The pretrained backbone already knows a lot. If our new dataset is small, letting all those millions of knobs wiggle would let them memorise noise. Freezing them = "keep what you know, just learn a new final decision." (In code this is requires_grad=False.)

PICTURE. The backbone is now drawn as a stone block — greyed, locked. Only the small head is "live" (red). Gradients flow back only into the head and stop at the wall.

$\frac{\partial \mathcal{L}}{\partial \varphi }\text{: computed and used}\frac{\partial \mathcal{L}}{\partial \theta }\text{: never computed}$

• Left term: the head's gradient — kept, so $\varphi$ updates each step.
• Right term: the backbone's gradient — thrown away (we don't even bother building it), so $\theta$ is frozen.

---

Step 4 — Full fine-tuning: let the whole machine flow

WHAT. Now nothing is frozen. Every knob — backbone $\theta$ and head $\varphi$ — receives a gradient and moves.

WHY? If your data is large and different from the pretraining world (e.g. legal text vs everyday text), the frozen features are simply wrong for you. You need the backbone itself to reshape its features. Only unfreezing lets it do that.

PICTURE. The stone block turns back into a living network (red, unlocked). The gradient — the red backward arrow — now flows all the way from the loss on the right, through the head, and deep into every layer of the backbone on the left.

Both updates now happen:

$\theta \leftarrow \theta -\eta \frac{\partial \mathcal{L}}{\partial \theta },\varphi \leftarrow \varphi -\eta \frac{\partial \mathcal{L}}{\partial \varphi }$

To get the backbone gradient $\frac{\partial \mathcal{L}}{\partial \theta }$ we must use the chain rule — a way to multiply "how much each stage affects the next" along the whole chain:

$\frac{\partial \mathcal{L}}{\partial \theta }=\underset{\begin{array}{c}\text{loss}\\ \text{vs prediction}\end{array}}{\underbrace{\frac{\partial \mathcal{L}}{\partial \hat{y}}}}\cdot \underset{\begin{array}{c}\text{head vs}\\ \text{features}\end{array}}{\underbrace{\frac{\partial h_{\varphi }}{\partial f_{\theta }}}}\cdot \underset{\begin{array}{c}\text{features vs}\\ \text{backbone knobs}\end{array}}{\underbrace{\frac{\partial f_{\theta }}{\partial \theta }}}$

---

Step 5 — The cost gap you can now count

WHAT. Count the knobs that actually move in each strategy.

WHY? Because memory and compute scale with trainable knobs, not total knobs. This count is the practical difference.

PICTURE. Two bars: a tiny red sliver (feature extraction, only the head) beside a towering red bar (full FT, backbone + head). Same total height of "existing knobs," wildly different "moving knobs."

$P_{\text{trainable}}^{\text{feat}}=\underset{\text{a few thousand}}{\underbrace{P_{\text{head}}}}{P}_{\text{trainable}}^{\text{full}}=\underset{\text{millions to billions}}{\underbrace{P_{\text{backbone}}+P_{\text{head}}}}$

Now the memory. A common optimizer (Adam) stores, per moving knob: the knob, its gradient, and two "momentum" numbers $m,v$ — four numbers total:

$\text{Mem}\approx \underset{\begin{array}{c}\text{knob}+\text{grad}\\ +m+v\end{array}}{\underbrace{4}}\times \underset{\text{moving knobs}}{\underbrace{P_{\text{trainable}}}}\times (\text{bytes per number})$

• Full FT: $P_{\text{trainable}}$ is everything → this term explodes.
• Feature extraction: $P_{\text{trainable}}$ is tiny → this term nearly vanishes; you pay only for a single forward pass through the backbone (no expensive backward trip).

That is the whole trade-off, made of nothing but the update rule from Step 2.

---

Step 6 — Edge case: the small learning rate in full FT

WHAT. When we do unfreeze, we shrink $\eta$ to something tiny like $2\times 10^{-5}$, far smaller than the rate used during pretraining.

WHY? Look back at Step 2's landscape. The pretrained model already sits near the bottom of a good valley. A big step $\eta$ would fling the ball out of that valley — the backbone would overwrite its hard-won knowledge. This is catastrophic forgetting.

PICTURE. Same valley, two red arrows from the same starting ball. The big step leaps across the valley and lands higher (worse, knowledge lost). The small step gently slides toward the bottom (better, knowledge kept).

---

Step 7 — The degenerate middle: partial unfreezing

WHAT. Between "all frozen" and "all free" lies unfreezing only the top $k$ layers.

WHY? Layers are not equal. Early (shallow) layers learn general patterns worth keeping; later (deep) layers learn task-specific patterns worth adapting. So freeze the general ones, thaw the specific ones — see Layer-wise Representations in Deep Nets.

PICTURE. The stacked network: bottom layers still stone (frozen, black), top few layers now red (unfrozen). The gradient flows down only until it hits the stone wall partway.

---

The one-picture summary

Everything above collapses into one diagram: the same two-stage machine, drawn three times along a dial from frozen to free, with the moving (red) region growing and the cost/flexibility both rising.

Recall Feynman: retell the whole walkthrough

A model is a chef (backbone) plus a final serving instruction (head). Learning means nudging each of the chef's habits a little every time a dish is wrong — the "nudge direction" is the gradient, and the "nudge size" is the learning rate $\eta$. Feature extraction: freeze the chef into a statue. His cooking never changes, so you can cook every dish once and save it. You only teach the tiny serving step. Cheap, safe on little data — but you're stuck with the chef's existing taste. Full fine-tuning: wake the chef up; let every habit change. Now he can learn a whole new cuisine — but you must send feedback all the way back through every habit (the chain rule), which is slow and memory-hungry, and if you nudge too hard (big $\eta$) he forgets his old recipes (catastrophic forgetting). In between: wake only the chef's newest habits (top layers) and keep the basics frozen. It's a dial, and how far you turn it depends on how much data you have.

Recall Quick self-test

Which term becomes zero in feature extraction? ::: $\frac{\partial \mathcal{L}}{\partial \theta }$ — the backbone gradient. Why can you cache features when frozen but not when fine-tuning? ::: Frozen $\theta$ ⇒ $f_{\theta }(x)$ is constant; full FT keeps changing $\theta$ so features shift every step. Why a tiny $\eta$ in full FT? ::: The model starts near a good optimum; a big step overshoots and causes catastrophic forgetting. What sets where you land on the frozen↔free dial? ::: Dataset size and domain shift — a bias–variance tradeoff.

Related: Transfer Learning · Catastrophic Forgetting · PEFT and LoRA · Bias-Variance Tradeoff · Learning Rate Schedules · Layer-wise Representations in Deep Nets