Foundations — Full fine-tuning vs feature extraction

This page assumes you have seen nothing. We will earn every symbol before using it.

0. The picture behind everything: a model is a pipeline

Before any symbol, hold this image: data flows left to right through boxes, each box transforming numbers into more useful numbers.

Look at the figure. The input $x$ (say, a sentence) enters on the left. It passes through a long chain of processing boxes — that whole chain is the backbone. The backbone spits out a compact list of numbers called a feature vector (the amber arrow). A final small box, the head, reads that vector and produces the answer $\hat{y}$ (e.g. "positive" or "negative").

Everything in this topic is about which boxes we allow to change during training. Keep this pipeline in your mind's eye — every symbol below labels one part of it.

1. $x$ — the input

Picture: the leftmost box in Figure s01, the arrow labelled $x$.

Why the topic needs it: every gradient and feature we discuss is computed for a specific input $x$. Without an input there is nothing to process.

2. Numbers-as-vectors: what "a vector of features" means

Picture: think of a row of little gauges, each showing one number. A 768-d vector is 768 gauges side by side.

Why the topic needs it: the backbone's output — the thing the head reads — is exactly such a vector. When the parent says "the 768-d CLS vector," it means "a list of 768 numbers summarising the sentence."

3. Weights and parameters — the knobs inside the boxes

Picture: imagine each box in Figure s01 has a control panel bristling with dials — some dials scale the inputs (weights), some slide a baseline up or down (biases). Training = a machine that slowly rotates all of them to make the final answer better.

Why the topic needs it: the entire feature-extraction-vs-full-fine-tuning debate is: which dials do we let turn? Freeze = lock the dials. Unfreeze = free the dials.

4. $\theta$ and $\varphi$ — naming the two groups of knobs

Picture: in Figure s01, colour the backbone's dials one colour and call the whole set $\theta$; colour the head's dials another and call that set $\varphi$.

Why the topic needs it: the parent writes updates separately for $\theta$ and $\varphi$ because the whole point is treating them differently — freeze $\theta$, train $\varphi$, or train both.

5. $f_{\theta }$ and $h_{\varphi }$ — functions: "give me an input, I give you an output"

Picture: $f_{\theta }$ is the whole backbone box; the subscript $\theta$ says "this box's behaviour depends on those dials." Turn the dials → the box computes something different.

Picture of the nesting: the parentheses are boxes-inside-boxes. $f_{\theta }(x)$ is computed first (inner box), then handed to $h_{\varphi }$ (outer box). This matches the flow in Figure s01 exactly.

Why the topic needs it: this one equation is the whole setup. "Feature extraction" freezes $f_{\theta }$; "full fine-tuning" lets $f_{\theta }$ change too. See Layer-wise Representations in Deep Nets for what those intermediate features actually represent.

6. $\hat{y}$, $y$, and $\mathcal{L}$ — prediction, truth, and how wrong we are

Picture: an archery target. $y$ is the bullseye. $\hat{y}$ is where the arrow landed. $\mathcal{L}$ is the distance between them — the miss.

Why the topic needs it: training means "turn the dials to make $\mathcal{L}$ smaller." Every gradient in the parent note is a gradient of this loss. No loss, no direction to improve.

7. The derivative $\frac{\partial \mathcal{L}}{\partial w}$ — "which way should I turn this knob?"

This is the one piece of real calculus. We build it from zero.

Picture: stand on a hillside where height = loss. The derivative is the steepness of the slope directly under your foot for knob $w$. To descend, step opposite to the uphill direction.

Picture: in Figure s03, $\eta$ is the length of the downhill arrow. Too long and you leap over the valley; too short and you crawl. This is exactly why the parent insists on a small $\eta$ for full fine-tuning — big steps here would smash the carefully-learned dials. More at Learning Rate Schedules.

Why the topic needs it: "backbone gradients: not computed" (feature extraction) literally means we never calculate $\partial \mathcal{L}/\partial \theta$, so those $\theta$ dials never move.

8. The chain rule — why the backbone gradient is a product

Picture: a relay race passing a baton (the signal) backwards through the pipeline of Figure s01 — from the loss at the right end all the way to the deep backbone dials on the left. Passing this baton through the whole backbone is the expensive backward pass.

Why the topic needs it: this is the reason full fine-tuning is heavy — you must run the baton through every box. Feature extraction stops the baton at the head, so it's cheap.

9. Freeze / unfreeze — what those words do to the equations

Picture: put padlocks on the backbone dials in Figure s01. Locked = frozen. The signal still flows forward (you still get features), but no dial rotates.

10. Counting knobs and memory — $P$ and the Adam factor

Why the topic needs it: this is the concrete "cost" side of the "flexibility vs cost" trade-off. The whole Bias-Variance Tradeoff and PEFT and LoRA discussion rides on this count.

How the foundations feed the topic

Read the map top to bottom: the pipeline produces a prediction, the loss measures error, the derivative tells each knob which way to turn, the chain rule carries that signal into the backbone, and the decision to freeze or unfreeze $\theta$ is the topic itself.

Equipment checklist

Test yourself — reveal only after answering.

Return to the parent: Full fine-tuning vs feature extraction · Prerequisite: Transfer Learning.