Transfer learning and fine-tuning

3.4.11 · AI-ML › Convolutional Neural Networks

Transfer learning neural networks ke liye exactly yahi karta hai: hum ek model lete hain jo ek massive dataset par train kiya gaya hai (jaise ImageNet ke 14M images) aur uske seekhe hue features ko apne chhote task ke liye repurpose karte hain (jaise 1000 medical scans classify karna). Early layers ne pehle se hi universal patterns seekh liye hain—edges, textures, shapes—toh hum unhe scratch se dobara seekhne mein waqt barbaad nahi karte.

Why Transfer Learning Works

Key insight yeh hai: Deep CNs hierarchical features seekhte hain:

• Early layers (input ke paas): Generic low-level features (edges, colors, simple textures)
• Middle layers: Mid-level patterns (object parts, textures combinations)
• Late layers (output ke paas): Task-specific high-level concepts (specific object classes)

Early layers remarkably transferable hote hain kyunki edges aur textures almost har visual task mein appear karte hain. Ek network jo cats aur dogs mein distinguish karna seekha hai usne pehle hi "vertical edge kya hota hai" aur "fur texture kya hota hai" seekh liya hai—yeh knowledge medical imaging, satellite analysis, ya kisi bhi aur vision task ke liye useful hai.

Standard training dataset $\mathcal{D}$ par loss $\mathcal{L}$ minimize karta hai: ${\theta }^{\ast }=\arg {\min }_{\theta }{\mathbb{E}}_{(x,y)\sim \mathcal{D}}[\mathcal{L}(f_{\theta }(x),y)]$

Limited data ke saath ($|\mathcal{D}|\ll$ millions), yeh optimization severely underconstrained hoti hai—kai parameter configurations $\theta$ chhote training set ko fit karti hain lekin poorly generalize karti hain (high variance).

Transfer learning model ko decompose karta hai: $f_{\theta }(x)=g_{{\theta }_{\text{head}}}(h_{{\theta }_{\text{base}}}(x))$

jahaan:

• $h_{{\theta }_{\text{base}}}$ = feature extractor (large source dataset ${\mathcal{D}}_{\text{source}}$ par pretrained)
• $g_{{\theta }_{\text{head}}}$ = task-specific head (chhote target dataset ${\mathcal{D}}_{\text{target}}$ par train kiya gaya)

Yeh sample complexity kyun reduce karta hai:

1. ${\theta }_{\text{base}}$ ek aise point par initialize hota hai jo pehle se ek achhe local minimum ke paas hai (${\mathcal{D}}_{\text{source}}$ se seekha gaya)
2. Hum sirf parameters ka ek fraction optimize karte hain (${\theta }_{\text{head}}$ typically total ka 0.1-10%)
3. Effective model capacity frozen base se constrained hoti hai, jo strong regularization ka kaam karta hai

$\text{Sample complexity}\propto \frac{\text{\# trainable parameters}}{\text{dataset size}}$

Zyaadatar parameters ko freeze karke, hum is ratio ki left side ko drastically reduce karte hain.

The Three Transfer Learning Strategies

Strategy 1: Feature Extraction (Frozen Base)

Kab use karna hai:

• Bahut chhota target dataset (< 1000 samples)
• Target task source task se similar ho
• Limited compute budget ho

Yeh kaise kaam karta hai:

# Pseudo-code
base_model = load_pretrained_model("ResNet50", weights="imagenet")
base_model.trainable = False  # Freeze all base layers
 
# Add custom head
model = Sequential([
    base_model,
    GlobalAveragePooling2D(),
    Dense(256, activation='relu'),
    Dropout(0.5),
    Dense(num_classes, activation='softmax')
])
 
# Only head parameters update
model.compile(optimizer=Adam(lr=1e-3))

Derivation ki yeh kyun kaam karta hai:

Pretrained features $h_{{\theta }_{\text{base}}}(x)$ ek representation space form karte hain. Hum is space se target labels tak ek linear (ya shallow nonlinear) mapping seekh rahe hain:

$\hat{y}=\text{softmax}(W{h}_{{\theta }_{\text{base}}}(x)+b)$

Yeh raw pixels se poori mapping seekhne se kaafi aasaan hai, kyunki $h(x)$ pehle se hi relevant structure capture kar chuka hai. Optimization convex hoti hai (agar sirf linear head use kar rahe hain) ya low-dimensional (agar ek hidden layer add kar rahe hain).

Strategy 2: Fine-tuning (Partial Unfreezing)

Kab use karna hai:

• Medium dataset (1k-100k samples)
• Target task source se moderately different ho
• Tumhare paas full training ke liye compute ho

Step-by-step yeh kaise kaam karta hai:

Step 1: Head ko frozen base ke saath train karo (jaise feature extraction mein) kuch epochs ke liye. Isse randomly-initialized head ek reasonable state mein aa jaata hai.

Step 2: Last few convolutional blocks unfreeze karo:

# Unfreeze last 2 blocks (out of 5) ResNet50
for layer in base_model.layers[-30:]:  # ~2 blocks
    layer.trainable = True

Step 3: Bahut chhoti learning rate ke saath training jaari rakho: ${\text{lr}}_{\text{fine-tune}}=\frac{{\text{lr}}_{\text{initial}}}{10}\text{ to }\frac{{\text{lr}}_{\text{initial}}}{100}$

Chhoti learning rate kyun? Gradient update perspective se derive karte hain:

Pretrained weights ${\theta }_{\text{base}}^{\text{pretrain}}$ pehle se source task ke liye ek achhe local minimum par hain. Badi learning rate $\eta$ ke saath, gradient update:

${\theta }_{\text{base}}^{t+1}={\theta }_{\text{base}}^t-\eta {\nabla }_{{\theta }_{\text{base}}}{\mathcal{L}}_{\text{target}}$

is achhe initialization se door ja sakta hai aur useful features destroy kar sakta hai (catastrophic forgetting).

Chhoti $\eta$ ensure karti hai: $\Vert {\theta }_{\text{base}}^{t+1}-{\theta }_{\text{base}}^{\text{pretrain}}\Vert \ll 1$

Hum pretrained solution ke aas-paas basin of attraction mein rehte hain, sirf target task ke liye chhote adjustments karte hain.

Differential learning rates (layer-wise):