Full fine-tuning vs feature extraction: Question bank

This page assumes you have read the parent note. Terms like freeze, head, backbone, and catastrophic forgetting are used as defined there. If any feel shaky, re-read the parent first — here we only stress-test them.

True or false — justify

Each item is a statement. Decide true/false, then give the reason. The reason is the point, not the verdict.

More trainable parameters always means better downstream accuracy.

False — extra capacity only helps if you have enough data to constrain it; on a small set the free parameters fit noise, so accuracy on new data drops (overfitting).

In feature extraction the backbone still runs a forward pass on every input.

True — you must compute the representation

f_{θ} (x)

to feed the head; you just skip the backward pass and the weight updates through

θ

Feature extraction and full fine-tuning produce the same internal features throughout training.

False — full FT updates

θ

, so

f_{θ} (x)

drifts every step; feature extraction holds

θ

fixed, so

f_{θ} (x)

is a constant vector for each input.

You can cache the backbone's outputs once and reuse them across all epochs in feature extraction.

True — a frozen backbone maps each input to the same vector forever, so one pass over the data gives embeddings you can train the head on repeatedly (Transfer Learning leans on exactly this).

You can cache backbone outputs once in full fine-tuning too, to save time.

False — full FT changes

θ

every step, so

f_{θ} (x)

changes too; a cached vector would be stale after the first update.

Using a small learning rate is only about numerical stability.

False — the main reason is to nudge the already-good pretrained weights gently so you don't overwrite general knowledge (Catastrophic Forgetting); stability is a secondary bonus.

Freezing the backbone means you never have to think about its dropout or BatchNorm mode.

False — dropout and BatchNorm behave differently in train() vs eval(); a frozen backbone left in train mode still perturbs/updates running stats, shifting your "fixed" features.

PEFT methods like LoRA are just a fancy name for full fine-tuning.

False — PEFT and LoRA keeps almost all backbone weights frozen and trains a small set of injected parameters, sitting near the feature-extraction end of the spectrum on trainable-param count.

Feature extraction gives strong regularization "for free".

True — freezing the backbone drops the trainable-parameter count by orders of magnitude, which shrinks the hypothesis space and pushes you toward the low-variance side of the Bias-Variance Tradeoff.

Spot the error

Each line contains a flawed statement or plan. Name what's wrong.

"My dataset is 300 examples, so I'll fully fine-tune all 110M params for maximum accuracy."

110M free params over 300 examples is extreme high variance — it will memorize the tiny set; freeze the backbone and train only the head instead.

"I'll reuse the pretraining learning rate (

1 0^{- 3}

) for fine-tuning because it clearly worked before."

The model now sits near a good optimum; a large LR takes big steps that overwrite learned features (catastrophic forgetting). Use

1 0^{- 5}

–

1 0^{- 4}

or Learning Rate Schedules with warmup.

"Feature extraction is cheap, so it needs no optimizer memory at all."

The head still trains, so it still needs gradients and optimizer state — but only for the tiny head, which is why the cost nearly vanishes, not fully.

"I froze the backbone, so I set the whole model to train() and moved on."

The frozen backbone should be in eval() so its BatchNorm uses fixed running stats and dropout is off; otherwise your "fixed" features silently shift between batches.

"Domain shift is huge and I have 500k labels, so I'll do linear probing to be safe."

Frozen features from a different domain are the wrong features — that is high bias, not a variance problem. With this much data, full fine-tuning is the right move.

"Partial fine-tuning means unfreezing the first few (shallow) layers."

It's the opposite — shallow layers hold general features worth keeping frozen; the later layers are most task/domain-specific, so you usually unfreeze the top

k

(Layer-wise Representations in Deep Nets).

"Since feature extraction avoids the backward pass through the backbone, it also avoids the forward pass."

The forward pass is mandatory to produce the representation; only the backbone's backward pass and weight update are skipped.

"Adam stores just the weight and gradient, so full FT memory is

2 \times P

Adam also stores two moment estimates

m

and

v

per parameter, so budget roughly

4 \times P_{trainable}

numbers — a big deal when

P_{trainable}

is the whole backbone.

Why questions

Answer the mechanism, not just the label.

Why does feature extraction resist overfitting on small data?

Only the head's parameters can move, so the effective model complexity is tiny; a small hypothesis space can't contort itself around noise in a small dataset.

Why does full fine-tuning demand so much more memory than feature extraction?

It computes and stores gradients and optimizer state (Adam's

m, v

) for every backbone weight, and must hold intermediate activations for the full backward pass through the huge backbone.

Why can a small learning rate prevent catastrophic forgetting?

Small steps keep the updated weights close to their pretrained values, so the general knowledge encoded in

θ

is perturbed only slightly while the task head still steers adaptation.

Why do we often use discriminative (layer-wise) learning rates in full FT?

Shallow layers hold general features that should barely move, while later layers need more adaptation — so a smaller LR deep and larger LR shallow protects reusable knowledge while letting task-specific layers change (Learning Rate Schedules).

Why is feature extraction attractive when serving many downstream tasks?

You can share one frozen backbone across all tasks and only swap tiny per-task heads, saving both memory and the cost of storing many full model copies.

Why does full fine-tuning need a full backward pass while feature extraction does not?

To update

θ

you must evaluate the chain

\frac{\partial L}{\partial θ} = \frac{\partial L}{\partial y ^} \frac{\partial h _{ϕ}}{\partial f _{θ}} \frac{\partial f _{θ}}{\partial θ}

all the way through the backbone; feature extraction discards

\frac{\partial L}{\partial θ}

, so it stops the backward pass at the head.

Why can feature extraction be "capped" in accuracy even with unlimited head training?

The head can only combine whatever the frozen features already encode; if the pretrained representation lacks the distinctions your task needs, no head can recover them.

Edge cases

The boundary scenarios where the simple rules bend.

Dataset is huge but the task domain is nearly identical to pretraining — which strategy?

Frozen features are already well-matched (low bias), so feature extraction can match full FT at a fraction of the cost; full FT buys little here.

Dataset is tiny but the domain is wildly different from pretraining.

The painful case — full FT overfits (tiny data) yet frozen features are wrong (big shift). Compromise with PEFT/LoRA or unfreezing only the top layer with heavy regularization.

What happens to the frozen features during a feature-extraction run — do they move at all?

No —

θ

is constant, so each input's

f_{θ} (x)

is identical every epoch; only the head's decision boundary over those fixed points changes.

Backbone learning rate set to exactly zero in a "full FT" setup — what have you actually built?

Feature extraction in disguise: with

η_{θ} = 0

the backbone never updates, so it behaves like a frozen feature computer regardless of whether gradients were computed.

Head learning rate set to zero while the backbone trains — is that useful?

Almost never — a randomly-initialized head produces meaningless gradients, so the backbone adapts toward a broken target; the head must learn (or be warmed up) for backbone updates to be meaningful.

Zero fine-tuning data (zero-shot) — where does that sit on the spectrum?

At the extreme past feature extraction: nothing trains at all, so you rely entirely on the pretrained model's built-in mapping with no head or weight updates.

You unfreeze all layers but keep the LR so small that weights barely move — outcome?

Effectively close to feature extraction in behaviour (features change negligibly) but you still pay full FT's memory and compute cost — the worst of both unless you plan to raise the LR later.

Recall Fastest self-check

If you can state, in one breath, (1) what is frozen, (2) whether embeddings are cacheable, (3) why the learning rate is small, and (4) which dataset size favours which strategy — you own this topic. If any answer is a bare yes/no, revisit the parent note.