Question bank — Transfer learning and fine-tuning

False — with a tiny target set the frozen base acts as strong regularization, so unfreezing can overfit and score lower; fine-tuning only wins when you have enough data to update those weights safely.

False — the early layers learn generic edges and textures that appear in nearly every visual task, so even unrelated targets (satellite, medical) benefit from the low-level features.

True — with the base fixed, the feature vector $h(x)$ is a constant list of numbers, so training $\text{softmax}(Wh(x)+b)$ is exactly a linear classifier, whose cross-entropy loss is a single convex bowl in $W,b$ with no false local minima.

False — the small rate keeps updates inside the basin of the pretrained minimum, so it changes which solution you converge to, preventing catastrophic forgetting.

Careful — freezing weights is not enough; you must also stop BN from updating its running mean/variance, or it will drift to your small dataset's statistics and corrupt the frozen features (see 3.4.7-Batch-normalization).

False — a small target set still overfits the trainable head; augmentation (see 4.1.3-Data-augmentation) remains one of the cheapest ways to lower variance.

False — sample complexity scales as $\frac{\#\text{trainable params}}{\text{dataset size}}$, so with little data, more trainable parameters raise variance and hurt generalization.

The head is random, so its large early gradients flow back and destroy the pretrained features before they can help; you should first warm up the head with the base frozen.

In most frameworks the trainable flag only takes effect after recompiling; without it the optimizer still holds the old (trainable) variable list.

A rate good for a random head is far too large for near-optimal pretrained weights — it overshoots the good minimum; use differential rates (small for base, large for head).

Backwards: early layers hold the most transferable generic features and should change least; the late task-specific layers are the ones worth unfreezing.

Flattening defeats the purpose of GAP; global average pooling already collapses each $7\times 7$ channel to one number, giving $2048$ features with no huge spatial FC layer.

The symptom is overfitting, and a smaller rate barely helps; better fixes are more regularization (3.2.5-Regularization-techniques), augmentation, or refreezing more layers.

Catastrophic forgetting: the large step drove weights far from the pretrained basin, wrecking the general features, not an architecture flaw.

So the randomly-initialized head reaches a sensible state before its gradients are allowed to reach and corrupt the fragile pretrained features.

They encode the most general, most reusable features (edges, colors), so they need the least task-specific adjustment and are the costliest to damage.

It shrinks the effective hypothesis space — fewer free parameters means fewer configurations that can memorize noise, lowering variance just like an explicit penalty.

Many parameter settings fit the few training points equally well, so the optimizer has no signal to pick the one that generalizes — high variance.

You begin near a good minimum, so far fewer gradient steps (and thus fewer labeled examples) are needed to reach a good solution than starting from random init.

The skip paths (3.4.8-Residual-connections) keep gradients flowing cleanly to unfrozen deep blocks, so those layers actually receive a useful signal instead of a vanished one.

Feature extraction (or light fine-tuning) — the tasks match, so the pretrained head is nearly correct; heavy unfreezing risks overfitting the small new-domain set.

Full fine-tuning with differential rates — big data supports updating everything, and the large domain gap means even mid/late features must be relearned.

No — with ~4 images per class, unfreezing invites severe overfitting; keep the base frozen and train only a small regularized head.

The advantage of transfer shrinks: with abundant data even random init can find a good minimum, so transfer mainly saves training time rather than final accuracy.

Transfer gives no benefit and can even hurt — the pretrained init sits in an irrelevant basin, so you're better off training the small parts from scratch.

Plain feature extraction — a zero step size means the base never updates, identical to freezing it.