Self-supervised and contrastive learning (SimCLR, CLIP)

6.5.4 · AI-ML › Research Frontiers & Practice

What is Self-supervised Learning?

Key distinction:

• Unsupervised learning: Structure dhundho (clustering, density estimation) bina labels ke
• Self-supervised learning: Data structure se pseudo-labels banao, phir unse train karo
• Supervised learning: Human-provided labels use karo

Why does this work?

Pretext tasks achhe se solve karne ke liye, model ko data ke baare mein general features sikhne padte hain. Agar tum ek model ko train karo ye predict karne ke liye ki kya do image crops ek hi photo se aate hain, to use seekhna hoga:

• Object parts aur boundaries
• Colors, textures, shapes
• Spatial relationships

Ye representations downstream tasks jaise classification ya detection mein transfer hoti hain.

What is Contrastive Learning?

The contrastive objective mathematically:

Anchor sample $x$ ke liye, positive $x^{+}$ (augmented/related view), aur negatives $\{x_i^{-}\}$:

${\mathcal{L}}_{\text{contrastive}}=-\log \frac{\exp (\text{sim}(z,z^{+})/\tau )}{\exp (\text{sim}(z,z^{+})/\tau )+{\sum }_{i=1}^N\exp (\text{sim}(z,z_i^{-})/\tau )}$

jahan:

• $z=f(x)$ seekha hua representation hai
• $\text{sim}(u,v)=\frac{u^{\top }v}{\Vert u\Vert \Vert v\Vert }$ cosine similarity hai
• $\tau$ ek temperature parameter hai (concentration control karta hai)

Ye form kyun? Ye InfoNCE loss hai (Noise Contrastive Estimation). Chalte hain derive karte hain kyun ye sense banata hai:

1. Goal: Positive pair ko similar banao, negatives ko dissimilar
2. Softmax over similarities: Ise N+1 choices mein se positive ko classify karne ki tarah treat karo
3. Temperature $\tau$: Chhota $\tau$ → sharper distribution, model ko zyada confident hona padega
4. Logarithm: Products ko sums mein convert karo, gradients stabilize karo

Denominator normalize karta hai taki loss interpret ho sake "positive sample identify karne ki negative log probability" ke roop mein.

SimCLR: Simple Framework for Contrastive Learning

The SimCLR Recipe (4 key ingredients)

1. Data Augmentation Image $x$ se do random augmented views ${\tilde{x}}_i$ aur ${\tilde{x}}_j$ banao:

• Random crop + resize
• Color distortion (brightness, contrast, saturation, hue)
• Gaussian blur
• Random grayscale

Kyun? Strong augmentation challenging positive pairs banata hai. Model pixels memorize karke cheat nahi kar sakta; use semantic features sikhne padte hain jo in transforms ke baad bhi invariant rahein.

2. Base Encoder Ek neural network $f(\cdot )$ (typically ResNet) jo images ko representations mein map karta hai: $h_i=f({\tilde{x}}_i)\in {\mathbb{R}}^d$

3. Projection Head Ek chhota MLP $g(\cdot )$ jo representations ko ek aisi space mein map karta hai jahan contrastive loss apply hota hai: $z_i=g(h_i)=W^{(2)}\sigma (W^{(1)}{h}_i)\in {\mathbb{R}}^{128}$

Projection head kyun? Loss $z$ par apply hota hai, lekin $h$ downstream tasks ke liye save karte hain. Ye separation performance improve karta hai (SimCLR paper mein empirically dikhaya gaya hai). Projection head mein pretext task ke liye useful information ho sakti hai jo transfer ke liye useful nahi hai.

4. Contrastive Loss $N$ samples ke ek batch ke liye, hume $2N$ augmented views milte hain. Har positive pair $(i,j)$ ke liye:

${\ell }_{i,j}=-\log \frac{\exp (\text{sim}(z_i,z_j)/\tau )}{{\sum }_{k=1}^{2N}{1}_{k\ne i}\exp (\text{sim}(z_i,z_k)/\tau )}$

Final loss saare positive pairs par average karta hai: $\mathcal{L}=\frac{1}{2N}{\sum }_{k=1}^N[{\ell }_{2k-1,2k}+{\ell }_{2k,2k-1}]$

Ye formulation kyun? Har augmented view ek baar anchor ki tarah serve karta hai. Hum batch ke baaki saare $2N-2$ samples ko negatives treat karte hain (batch ke saare images alag hain, sivaaye paired augmentation ke).

Step 1: 4 images sample karo: cat, dog, car, tree Step 2: Har ek ko do baar augment karo → 8 views: cat₁, cat₂, dog₁, dog₂, car₁, car₂, tree₁, tree₂

Step 3: $f$ aur $g$ se encode karo:

• $z_{{\text{cat}}_1},z_{{\text{cat}}_2},z_{{\text{dog}}_1},z_{{\text{dog}}_2},..$

Step 4: Anchor cat₁ ke liye loss compute karo:

• Positive: cat₂ (sim = 0.85)
• Negatives: dog₁, dog₂, car₁, car₂, tree₁, tree (sims ≈ 0.1-0.3)

${\ell }_{{\text{cat}}_1,{\text{cat}}_2}=-\log \frac{e^{0.85/0.5}}{e^{0.85/0.5}+e^{0.2/0.5}+e^{0.15/0.5}+...+e^{0.25/0.5}}$

$\tau =0.5$ ke saath, numerator $\approx e^{1.7}=5.47$, denominator $\approx 5.47+1.49+...\approx 15$, to loss $\approx -\log (5.47/15)\approx 1.01$.

Ye step kyun? Softmax positive pair ke contribution ko amplify karta hai. Gradients cat₁ aur cat₂ ko paas laate hain, negatives ko door dhakelte hain.

Result: Training ke baad, $h$ embeddings same objects ko saath cluster karte hain, naye images par generalize karte hain.

Why SimCLR Works: The Magic of Large Batches & Strong Augmentation

$\text{Negatives per sample}=2N-2$

$N=4096$ ke saath, har sample 8190 negatives ke against contrast karta hai. Ye rich supervision provide karta hai.

Temperature $\tau$: Negatives ki hardness control karta hai.

• Chhota $\tau$ (e.g., 0.1): Hard negatives par focus karo (sharp distribution)
• Bada $\tau$ (e.g., 1.0): Saare negatives ko zyada equally treat karo (soft distribution)

Typical value: $\tau \in [0.1,0.5]$

Augmentation strength: Stronger augmentation zyada transformations ke liye invariance force karta hai. SimCLR ne color distortion + crop + blur ka combination crucial paya.

CLIP: Contrastive Language-Image Pre-training

The CLIP Architecture

Do encoders:

1. Image encoder: Vision Transformer (ViT) ya ResNet $I:\text{image}\to {\mathbb{R}}^d$

2. Text encoder: Transformer $T:\text{caption}\to {\mathbb{R}}^d$

Dono dimension $d$ (typically 512 ya 1024) ke ek joint embedding space mein project karte hain.

Training data: Internet se scrape kiye gaye 400 million (image, text) pairs.

CLIP's Contrastive Objective: A Symmetric Dance

$N$ (image, caption) pairs ke ek batch ke liye:

1. Image embeddings compute karo: $\{I_1,I_2,...,I_N\}$
2. Text embeddings compute karo: $\{T_1,T_2,...,T_N\}$
3. Pairwise cosine similarities compute karo: $S_{ij}=\frac{I_i\cdot T_j}{\Vert I_i\Vert \Vert T_j\Vert }$

Symmetric loss: Har image ke liye, uska sahi caption positive hai, baaki saare negative hain (aur vice versa).

Image-to-text loss (row-wise softmax): ${\mathcal{L}}_{i2t}=-\frac{1}{N}{\sum }_{i=1}^N\log \frac{\exp (S_{ii}/\tau )}{{\sum }_{j=1}^N\exp (S_{ij}/\tau )}$

Text-to-image loss (column-wise softmax): ${\mathcal{L}}_{t2i}=-\frac{1}{N}{\sum }_{j=1}^N\log \frac{\exp (S_j/\tau )}{{\sum }_{i=1}^N\exp (S_{ij}/\tau )}$

Total CLIP loss: ${\mathcal{L}}_{\text{CLIP}}=\frac{1}{2}({\mathcal{L}}_{i2t}+{\mathcal{L}}_{t2i})$

Symmetric kyun? Dono directions (image→text aur text→image) gradient signal provide karte hain. Ye dono modalities ko equally treat karta hai, kisi ek ko dominate hone se rokta hai.

Similarity matrix (encoding ke baad, values cosine similarities hain):

         T_cat  T_dog  T_car  T_tree
I_cat     0.85   0.15   0.10   0.12
I_dog     0.18   0.82   0.14   0.11
I_car     0.09   0.13   0.88   0.15
I_tree    0.11   0.10   0.16   0.84

I_cat ke liye Image-to-text (T_cat pick karna hai): ${\ell }_{i2t}^{\text{cat}}=-\log \frac{e^{0.85/0.07}}{e^{0.85/0.07}+e^{0.15/0.07}+e^{0.10/0.07}+e^{0.12/0.07}}$

$\tau =0.07$ ke saath: numerator $\approx e^{12.1}\approx 1.8\times 10^5$, denominator correct pair se dominated, loss≈ 0.05 (chhota, achha).

Ye step kyun? High diagonal values (correct pairs) matlab low loss. Gradients encoders ko matching pairs aur strongly align karte hain.

Zero-shot use: Test time par, class "airplane" ke liye, image aur text "a photo of an airplane" ke beech similarity compute karo bina kisi airplane training label ke.

Why CLIP is Revolutionary

1. Zero-shot transfer: Naye tasks ke liye koi labeled data nahi chahiye

• Classes ke liye text prompts banao: "a photo of a [CLASS]"
• Image embedding ke saath similarity compute karo
• Sabse zyada similarity wali class predict karo

2. Robustness: Diverse internet data par train kiya gaya, curated datasets par nahi

• Distribution shift better handle karta hai
• Adversarial examples ke liye zyada robust (ImageNet quirks ke liye explicitly optimize nahi kiya)

3. Flexibility: Nuanced language samajhta hai

• "a photo of a cat" aur "a drawing of a cat" mein distinguish kar sakta hai
• Compositional queries handle karta hai: "a cat wearing sunglasses"

Encode karo: $I\to {\mathbb{R}}^d$, har $t_k\to {\mathbb{R}}^d$

Logits compute karo: ${\ell }_k=\frac{I\cdot t_k}{\Vert I\Vert \Vert t_k\Vert }\cdot \alpha$

jahan $\alpha =1/\tau$ ek learned temperature scaling hai.

Probability ke liye softmax apply karo: $p(y=k|I)=\frac{\exp ({\ell }_k)}{{\sum }_{j=1}^K\exp ({\ell }_j)}$

Ye kyun kaam karta hai: Embedding space ko is tarah train kiya gaya tha ki semantically matching (image, text) pairs ki cosine similarity high ho. Test time par, image naturally sahi class description ke saath align ho jaati hai.

Common Mistakes & Misconceptions

Ye galat kyun hai: Model kabhi ek hi augmented view do baar nahi dekhta. Use semantic features sikhne padte hain jo augmentations (rotation, crop, color jitter) ke baad bhi invariant rahein. Ye pixel patterns memorize nahi kar sakta.

Fix: Contrastive learning model ko force karta hai aise features extract karne ke liye jo augmentations ke liye robust hain. Ye features generalize karte hain kyunki ye high-level semantics capture karte hain, na ki low-level pixels.

Ye galat kyun hai: Contrastive learning ko bahut saare negatives chahiye. Batch size 32 ke saath, SimCLR ke paas sirf 62 negatives per sample hain. Ye weak signal provide karta hai.

Fix: Large batch sizes (≥1024) ya memory banks/momentum encoders (jaise MoCo) use karo negatives ka bada pool maintain karne ke liye. Zyada negatives → harder task → better representations.

Math: $N$ samples ke saath, negatives = $2N-2$. $N=32$ ke liye: 62 negatives. $N=4096$ ke liye: 8190 negatives (132× zyada contrast).

Ye galat kyun hai: Empirically, projection head $g(\cdot )$ use karna aur downstream tasks ke liye $h=f(x)$ save karna transfer performance 10+ percentage points improve karta hai.

Ye kyun kaam karta hai: Projection head mein contrastive pretext task ke liye useful information ho sakti hai (jaise color, exact spatial position) jo semantic downstream tasks ke liye useful nahi hai. Representation ($h$) ko contrastive space ($z$) se alag karke, hume better general features milte hain.

Ye galat kyun hai: Abundant labeled data wale tasks ke liye (jaise ImageNet classification), fine-tuned supervised models CLIP zero-shot se abhi bhi better perform karte hain. CLIP ki strength generalization hai naye tasks/distributions par, na ki standard benchmarks par peak accuracy.

Tradeoff: CLIP kuch peak accuracy flexibility aur robustness ke liye trade karta hai. CLIP use karo:

• Zero/few-shot learning ke liye
• Scarce labels wale tasks ke liye
• Domain adaptation ke liye
• Open-vocabulary recognition ke liye

Supervised fine-tuning use karo jab:

• Tumhare paas abundant labeled data ho
• Tumhe ek fixed task par maximum accuracy chahiye
• Latency/compute critical ho (CLIP models bade hote hain)

Connections

• Supervised learning fundamentals - contrastive learning expensive labeling se bachta hai
• Transfer learning and fine-tuning - self-supervised pretraining → downstream tasks par fine-tune karo
• Data augmentation techniques - SimCLR critically augmentation design par depend karta hai
• Metric learning and embedings - contrastive learning ek metric space sikhta hai semantic meaning ke saath
• Vision transformers (ViT) - CLIP image encoder ke roop mein ViT use karta hai
• Attention mechanisms and transformers - CLIP ka text encoder ek Transformer hai
• Zero-shot and few-shot learning - CLIP language ke zariye zero-shot classification enable karta hai
• Multi-modal learning - CLIP vision aur language modalities ko bridge karta hai
• Representation learning - core goal: aise features sikho jo tasks mein transfer ho sakein
• Batch normalization and training techniques - large batch training ko careful optimization chahiye (LARS optimizer)

Iske badle, tum ek game khelte ho: Main tumhe do photos dikhata hoon. Kabhi kabhi ye ek hi cat ki do pictures hoti hain (bas alag angles ya lighting se). Kabhi kabhi ek cat hoti hai aur ek dog. Tumhara kaam hai kehna "same" ya "different."

Pehle tum randomly guess karte ho. Lekin is game mein achhe hone ke liye, tum notice karne lagte ho: "Oh, cats ke pointy ears aur whiskers hote hain. Dogs ke floppy ears aur lambe snouts hote hain." Tum seekh rahe ho ki cats ko cats jaisa aur dogs ko dogs jaisa kya banata hai, chahe kisi ne unhe label nahi kiya!

Yahi hai contrastive learning. SimCLR ye laakhon images ke saath karta hai. Ye sikhta hai "ye do ajeeb dikhne wali pictures actually ek hi cat hain" camera tilt karne ya lighting change karne par kya nahi badalte upar focus karke.

CLIP language add karta hai: ab tumhare paas captions bhi hain. Ye sikhta hai "ye picture sentence 'a fluffy cat' ke saath jaati hai lekin 'a red car' ke saath nahi." To jab tum ise ek nayi picture dikhate ho aur poochte ho "kya ye cat hai ya car?", ye image ko un descriptions se compare karta hai aur sahi answer deta hai—bina training ke kabhi "ye ek cat photo hai" bataye!

CLIP = Contrastive Language-Image Pre-training "Images ko captions se CLIP karo contrastive loss se"

Contrastive loss mnemonic: "Negatives ko Push karo, Positives ko Pull karo" Loss denominator mein positives (numerator) aur negatives dono hain, normalized.

Key equation structure: $-\log (\text{positive}/(\text{positive}+\text{negatives}))$ Socho: "Ek lineup mein se positive pick karne ki probability maximize karo"

#flashcards/ai-ml

Contrastive learning ka core principle kya hai? :: Contrastive learning positive pairs ke beech agreement (similarity) maximize karta hai jabki negative pairs ke beech agreement ek learned embedding space mein minimize karta hai.

Anchor $z$, positive $z^{+}$, aur negatives $\{z_i^{-}\}$ ke liye InfoNCE contrastive loss formula likho :: $\mathcal{L}=-\log \frac{\exp (\text{sim}(z,z^{+})/\tau )}{\exp (\text{sim}(z,z^{+})/\tau )+{\sum }_i\exp (\text{sim}(z,z_i^{-})/\tau )}$ jahan $\text{sim}$ cosine similarity hai aur $\tau$ temperature hai.

Concept Map