Self-supervised and contrastive learning (SimCLR, CLIP)

What is Self-supervised Learning?

Key distinction:

• Unsupervised learning: Find structure (clustering, density estimation) with no labels
• Self-supervised learning: Create pseudo-labels from data structure, then train with those
• Supervised learning: Use human-provided labels

Why does this work?

To solve pretext tasks well, the model must learn general features about the data. If you train a model to predict whether two image crops come from the same photo, it must learn:

• Object parts and boundaries
• Colors, textures, shapes
• Spatial relationships

These representations transfer to downstream tasks like classification or detection.

What is Contrastive Learning?

The contrastive objective mathematically:

For anchor sample $x$, positive $x^+$ (augmented/related view), and negatives $\{x^-_i\}$:

$\mathcal{L}_{\text{contrastive}} = -\log \frac{\exp(\text{sim}(z, z^+) / \tau)}{\exp(\text{sim}(z, z^+) / \tau) + \sum_{i=1}^{N} \exp(\text{sim}(z, z^-_i) / \tau)}$

where:

• $z = f(x)$ is the learned representation
• $\text{sim}(u, v) = \frac{u^\top v}{\|u\| \|v\|}$ is cosine similarity
• $\tau$ is a temperature parameter (controls concentration)

Why this form? This is the InfoNCE loss (Noise Contrastive Estimation). Let's derive why it makes sense:

1. Goal: Make positive pair similar, negatives dissimilar
2. Softmax over similarities: Treat it as classifying the positive among N+1 choices
3. Temperature $\tau$: Smaller $\tau$ → sharper distribution, model must be more confident
4. Logarithm: Convert products to sums, stabilize gradients

The denominator normalizes so the loss interprets as "negative log probability of identifying the positive sample."

SimCLR: Simple Framework for Contrastive Learning

The SimCLR Recipe (4 key ingredients)

1. Data Augmentation Create two random augmented views $\tilde{x}_i$ and $\tilde{x}_j$ from image $x$:

• Random crop + resize
• Color distortion (brightness, contrast, saturation, hue)
• Gaussian blur
• Random grayscale

Why? Strong augmentation creates challenging positive pairs. The model can't cheat by memorizing pixels; it must learn semantic features invariant to these transforms.

2. Base Encoder A neural network $f(\cdot)$ (typically ResNet) that maps images to representations: $h_i = f(\tilde{x}_i) \in \mathbb{R}^d$

3. Projection Head A small MLP $g(\cdot)$ that maps representations to a space where contrastive loss is applied: $z_i = g(h_i) = W^{(2)} \sigma(W^{(1)} h_i) \in \mathbb{R}^{128}$

Why a projection head? Loss is applied to $z$, but we save $h$ for downstream tasks. This separation improves performance (empirically shown in the SimCLR paper). The projection head can contain information useful for the pretext task but not for transfer.

4. Contrastive Loss For a batch of $N$ samples, we get $2N$ augmented views. For each positive pair $(i, j)$:

$\ell_{i,j} = -\log \frac{\exp(\text{sim}(z_i, z_j) / \tau)}{\sum_{k=1}^{2N} \mathbb{1}_{k \neq i} \exp(\text{sim}(z_i, z_k) / \tau)}$

The final loss averages over all positive pairs: $\mathcal{L} = \frac{1}{2N} \sum_{k=1}^{N} [\ell_{2k-1, 2k} + \ell_{2k, 2k-1}]$

Why this formulation? Each augmented view serves as anchor once. We treat all other $2N-$ samples in the batch as negatives (all images in the batch are different, except the paired augmentation).

Step 1: Sample 4 images: cat, dog, car, tree
Step 2: Augment each twice → 8 views: cat₁, cat₂, dog₁, dog₂, car₁, car₂, tree₁, tree₂

Step 3: Encode through $f$ and $g$:

• $z_{\text{cat}_1}, z_{\text{cat}_2}, z_{\text{dog}_1}, z_{\text{dog}_2}, ..$

Step 4: Compute loss for anchor cat₁:

• Positive: cat₂ (sim = 0.85)
• Negatives: dog₁, dog₂, car₁, car₂, tree₁, tree (sims ≈ 0.1-0.3)

$\ell_{\text{cat}_1, \text{cat}_2} = -\log \frac{e^{0.85/0.5}}{e^{0.85/0.5} + e^{0.2/0.5} + e^{0.15/0.5} + ... + e^{0.25/0.5}}$

With $\tau=0.5$, numerator $\approx e^{1.7} = 5.47$, denominator $\approx 5.47 + 1.49 + ... \approx 15$, so loss $\approx -\log(5.47/15) \approx 1.01$.

Why this step? Softmax amplifies the positive pair's contribution. Gradients push cat₁ and cat₂ closer, push negatives away.

Result: After training, $h$ embedings cluster same objects together, generalize to new images.

Why SimCLR Works: The Magic of Large Batches & Strong Augmentation

$\text{Negatives per sample} = 2N - 2$

With $N=4096$, each sample contrasts against 8190 negatives. This provides rich supervision.

Temperature $\tau$: Controls hardness of negatives.

• Small $\tau$ (e.g., 0.1): Focus on hard negatives (sharp distribution)
• Large $\tau$ (e.g., 1.0): Treat all negatives more equally (soft distribution)

Typical value: $\tau \in [0.1, 0.5]$

Augmentation strength: Stronger augmentation forces invariance to more transformations. SimCLR found combining color distortion + crop + blur crucial.

CLIP: Contrastive Language-Image Pre-training

The CLIP Architecture

Two encoders:

1. Image encoder: Vision Transformer (ViT) or ResNet $I: \text{image} \rightarrow \mathbb{R}^d$

2. Text encoder: Transformer $T: \text{caption} \rightarrow \mathbb{R}^d$

Both project to a joint embedding space of dimension $d$ (typically 512 or 1024).

Training data: 400 million (image, text) pairs scraped from the internet.

CLIP's Contrastive Objective: A Symmetric Dance

Given a batch of $N$ (image, caption) pairs:

1. Compute image embedings: $\{I_1, I_2, ..., I_N\}$
2. Compute text embedings: $\{T_1, T_2, ..., T_N\}$
3. Compute pairwise cosine similarities: $S_{ij} = \frac{I_i \cdot T_j}{\|I_i\| \|T_j\|}$

The symmetric loss: For each image, its correct caption is positive, all others are negative (and vice versa).

Image-to-text loss (row-wise softmax): $\mathcal{L}_{i2t} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{ii} / \tau)}{\sum_{j=1}^{N} \exp(S_{ij} / \tau)}$

Text-to-image loss (column-wise softmax): $\mathcal{L}_{t2i} = -\frac{1}{N} \sum_{j=1}^{N} \log \frac{\exp(S_{j} / \tau)}{\sum_{i=1}^{N} \exp(S_{ij} / \tau)}$

Total CLIP loss: $\mathcal{L}_{\text{CLIP}} = \frac{1}{2}(\mathcal{L}_{i2t} + \mathcal{L}_{t2i})$

Why symmetric? Both directions (image→text and text→image) provide gradient signal. This treats both modalities equally, preventing one from dominating.

Similarity matrix (after encoding, values are cosine similarities):

         T_cat  T_dog  T_car  T_tree
I_cat     0.85   0.15   0.10   0.12
I_dog     0.18   0.82   0.14   0.11
I_car     0.09   0.13   0.88   0.15
I_tree    0.11   0.10   0.16   0.84

Image-to-text for I_cat (want to pick T_cat): $\ell_{i2t}^{\text{cat}} = -\log \frac{e^{0.85/0.07}}{e^{0.85/0.07} + e^{0.15/0.07} + e^{0.10/0.07} + e^{0.12/0.07}}$

With $\tau=0.07$: numerator $\approx e^{12.1} \approx 1.8 \times 10^5$, denominator dominated by correct pair, loss≈ 0.05 (small, good).

Why this step? High diagonal values (correct pairs) mean low loss. Gradients make encoders align matching pairs more strongly.

Zero-shot use: At test time, for class "airplane", compute similarity between image and text "a photo of an airplane" without any airplane training labels.

Why CLIP is Revolutionary

1. Zero-shot transfer: No labeled data for new tasks

• Create text prompts for classes: "a photo of a [CLASS]"
• Compute similarity with image embedding
• Predict class with highest similarity

2. Robustness: Trained on diverse internet data, not curated datasets

• Better handles distribution shift
• More robust to adversarial examples (not explicitly optimized for ImageNet quirks)

3. Flexibility: Understands nuanced language

• Can distinguish "a photo of a cat" vs "a drawing of a cat"
• Handles compositional queries: "a cat wearing sunglasses"

Encode: $I \rightarrow \mathbb{R}^d$, each $t_k \rightarrow \mathbb{R}^d$

Compute logits: $\ell_k = \frac{I \cdot t_k}{\|I\| \|t_k\|} \cdot \alpha$

where $\alpha = 1/\tau$ is a learned temperature scaling.

Apply softmax for probability: $p(y=k | I) = \frac{\exp(\ell_k)}{\sum_{j=1}^{K} \exp(\ell_j)}$

Why this works: The embedding space was trained so that semantically matching (image, text) pairs have high cosine similarity. At test time, the image naturally aligns with the correct class description.

Common Mistakes & Misconceptions

Why it's wrong: The model never sees the same augmented view twice. It must learn semantic features that remain invariant across augmentations (rotation, crop, color jitter). It can't memorize pixel patterns.

The fix: Contrastive learning forces the model to extract features robust to augmentations. These features generalize because they capture high-level semantics, not low-level pixels.

Why it's wrong: Contrastive learning needs many negatives. With batch size 32, SimCLR only has 62 negatives per sample. This provides weak signal.

The fix: Use large batch sizes (≥1024) or memory banks/momentum encoders (like MoCo) to maintain a large pool of negatives. More negatives → harder task → better representations.

Math: With $N$ samples, negatives = $2N-2$. For $N=32$:62 negatives. For $N=4096$: 8190 negatives (132× more contrast).

Why it's wrong: Empirically, using a projection head $g(\cdot)$ and saving $h=f(x)$ for downstream tasks improves transfer performance by 10+ percentage points.

Why it works: The projection head can contain information useful for the contrastive pretext task (e.g., color, exact spatial position) that's not useful for semantic downstream tasks. By separating representation ($h$) from contrastive space ($z$), we get better general features.

Why it's wrong: For tasks with abundant labeled data (e.g., ImageNet classification), fine-tuned supervised models still outperform CLIP zero-shot. CLIP's strength is generalization to new tasks/distributions, not peak accuracy on standard benchmarks.

The tradeoff: CLIP trades some peak accuracy for flexibility and robustness. Use CLIP for:

• Zero/few-shot learning
• Tasks with scarce labels
• Domain adaptation
• Open-vocabulary recognition

Use supervised fine-tuning when:

• You have abundant labeled data
• You need maximum accuracy on a fixed task
• Latency/compute is critical (CLIP models are larger)

Connections

• Supervised learning fundamentals - contrastive learning avoids expensive labeling
• Transfer learning and fine-tuning - self-supervised pretraining → fine-tune on downstream tasks
• Data augmentation techniques - SimCLR depends critically on augmentation design
• Metric learning and embedings - contrastive learning learns a metric space with semantic meaning
• Vision transformers (ViT) - CLIP uses ViT as image encoder
• Attention mechanisms and transformers - CLIP's text encoder is a Transformer
• Zero-shot and few-shot learning - CLIP enables zero-shot classification via language
• Multi-modal learning - CLIP bridges vision and language modalities
• Representation learning - core goal: learn features that transfer across tasks
• Batch normalization and training techniques - large batch training requires careful optimization (LARS optimizer)

Instead, you play a game: I show you two photos. Sometimes they're two pictures of the same cat (just from different angles or lighting). Sometimes one is a cat and one is a dog. Your job is to say "same" or "different."

At first you guess randomly. But to get good at this game, you start noticing: "Oh, cats have pointy ears and whiskers. Dogs have floppy ears and longer snouts." You're learning what makes cats look like cats and dogs look like dogs, even though nobody labeled them!

That's contrastive learning. SimCLR does this with millions of images. It learns "these two weird-looking pictures are actually the same cat" by focusing on what doesn't change when you tilt the camera or change the lighting.

CLIP adds language: now you also have captions. It learns "this picture goes with the sentence 'a fluffy cat' but not with'a red car'." So when you show it a new picture and ask "is this a cat or a car?", it compares the image to those descriptions and answers correctly—without ever being told "this is a cat photo" during training!

CLIP = Contrastive Language-Image Pre-training
"CLIP images to captions with contrastive loss"

Contrastive loss mnemonic: "Push Negatives, Pull Positives"
The loss denominator includes positives (numerator) and negatives, normalized.

Key equation structure: $-\log(\text{positive} / (\text{positive} + \text{negatives}))$
Think: "Maximize probability of picking the positive from a lineup"

#flashcards/ai-ml

What is the core principle of contrastive learning? :: Contrastive learning maximizes agreement (similarity) between positive pairs while minimizing agreement between negative pairs in a learned embedding space.

Write the InfoNCE contrastive loss formula for anchor $z$, positive $z^+$, and negatives $\{z_i^-\}$ :: $\mathcal{L} = -\log \frac{\exp(\text{sim}(z,z^+)/\tau)}{\exp(\text{sim}(z,z^+)/\tau) + \sum_i \exp(\text{sim}(z,z_i^-)/\tau)}$ where $\text{sim}$ is cosine similarity and $\tau$ is temperature.

Concept Map

Hinglish (regional understanding)