Transfer learning and fine-tuning

Transfer learning does exactly this for neural networks: we take a model trained on a massive dataset (like ImageNet's 14M images) and repurpose its learned features for our smaller task (like classifying 1000 medical scans). The early layers have already learned universal patterns—edges, textures, shapes—so we don't waste time relearning them from scratch.

Why Transfer Learning Works

The key insight: Deep CNs learn hierarchical features:

• Early layers (near input): Generic low-level features (edges, colors, simple textures)
• Middle layers: Mid-level patterns (object parts, textures combinations)
• Late layers (near output): Task-specific high-level concepts (specific object classes)

Early layers are remarkably transferable because edges and textures appear in nearly every visual task. A network trained to distinguish cats from dogs has already learned "what is a vertical edge" and "what is fur texture"—knowledge useful for medical imaging, satellite analysis, or any other vision task.

Standard training minimizes loss $\mathcal{L}$ over dataset $\mathcal{D}$: $\theta^* = \arg\min_{\theta} \mathbb{E}_{(x,y) \sim \mathcal{D}} [\mathcal{L}(f_\theta(x), y)]$

With limited data ($|\mathcal{D}| \ll$ millions), this optimization is severely underconstrained—many parameter configurations $\theta$ fit the small training set but generalize poorly (high variance).

Transfer learning decomposes the model: $f_\theta(x) = g_{\theta_{\text{head}}}(h_{\theta_{\text{base}}}(x))$

where:

• $h_{\theta_{\text{base}}}$ = feature extractor (pretrained on large source dataset $\mathcal{D}_{\text{source}}$)
• $g_{\theta_{\text{head}}}$ = task-specific head (trained on small target dataset $\mathcal{D}_{\text{target}}$)

Why this reduces sample complexity:

1. $\theta_{\text{base}}$ is initialized at point already near a good local minimum (learned from $\mathcal{D}_{\text{source}}$)
2. We only optimize a fraction of parameters ($\theta_{\text{head}}$ typically 0.1-10% of total)
3. The effective model capacity is constrained by the frozen base, acting as strong regularization

$\text{Sample complexity} \propto \frac{\text{\# trainable parameters}}{\text{dataset size}}$

By freezing most parameters, we drastically reduce the left side of this ratio.

The Three Transfer Learning Strategies

Strategy 1: Feature Extraction (Frozen Base)

When to use:

• Very small target dataset (< 1000 samples)
• Target task similar to source task
• Limited compute budget

How it works:

# Pseudo-code
base_model = load_pretrained_model("ResNet50", weights="imagenet")
base_model.trainable = False  # Freeze all base layers
 
# Add custom head
model = Sequential([
    base_model,
    GlobalAveragePooling2D(),
    Dense(256, activation='relu'),
    Dropout(0.5),
    Dense(num_classes, activation='softmax')
])
 
# Only head parameters update
model.compile(optimizer=Adam(lr=1e-3))

Derivation of why this works:

The pretrained features $h_{\theta_{\text{base}}}(x)$ form a representation space. We're learning a linear (or shallow nonlinear) mapping from this space to target labels:

$\hat{y} = \text{softmax}(W h_{\theta_{\text{base}}}(x) + b)$

This is much easier than learning the entire mapping from raw pixels, because $h(x)$ already captures relevant structure. The optimization is convex (if using only a linear head) or low-dimensional (if adding one hidden layer).

Strategy 2: Fine-tuning (Partial Unfreezing)

When to use:

• Medium dataset (1k-100k samples)
• Target task moderately different from source
• You have compute for full training

How it works step-by-step:

Step 1: Train head with frozen base (as in feature extraction) for a few epochs. This brings the randomly-initialized head to a reasonable state.

Step 2: Unfreeze the last few convolutional blocks:

# Unfreeze last 2 blocks (out of 5) ResNet50
for layer in base_model.layers[-30:]:  # ~2 blocks
    layer.trainable = True

Step 3: Continue training with much smaller learning rate: $\text{lr}_{\text{fine-tune}} = \frac{\text{lr}_{\text{initial}}}{10} \text{ to } \frac{\text{lr}_{\text{initial}}}{100}$

Why the small learning rate? Derive from gradient update perspective:

The pretrained weights $\theta_{\text{base}}^{\text{pretrain}}$ are already at a good local minimum for the source task. With large learning rate $\eta$, the gradient update:

$\theta_{\text{base}}^{t+1} = \theta_{\text{base}}^{t} - \eta \nabla_{\theta_{\text{base}}} \mathcal{L}_{\text{target}}$

could overshoot and move far from this good initialization, destroying useful features (catastrophic forgetting).

Small $\eta$ ensures: $\|\theta_{\text{base}}^{t+1} - \theta_{\text{base}}^{\text{pretrain}}\| \ll 1$

We stay in the basin of attraction around the pretrained solution, only making small adjustments for the target task.

Differential learning rates (layer-wise):