Foundations — Self-supervised and contrastive learning (SimCLR, CLIP)

This page is the toolbox. The parent note throws around symbols like $z$, $\text{sim}$, $\tau$, $\exp$, $\log$, and $\sum$ as if you already own them. Here we earn each one from nothing, in an order where every symbol only uses symbols we already built.

0. The picture that holds everything: the embedding map

Before any symbol, hold this image in your head.

Every image (or sentence) becomes a single dot on a flat map. The whole game is where the dots land. Two crops of the same cat should be neighbours; a cat and a car should be strangers on opposite sides of town. That map is called an embedding space, and each dot is an embedding (also called a representation).

Keep the map in mind — every symbol below is a tool for either placing a dot or measuring the distance between two dots.

1. A vector: the dot's address

To say where a dot is, we need coordinates.

The parent writes $z\in {\mathbb{R}}^{128}$. Read that as: "$z$ is a vector — a point — that lives in ($\in$) the space ${\mathbb{R}}^{128}$ of all lists of 128 real numbers." $\mathbb{R}$ is just the shorthand name for "all ordinary numbers" (the Reals: $3$, $-0.5$, $\pi$, ...). The little superscript counts how many coordinates.

2. A function: the machine that makes the dot

Something has to turn a photo into that list of numbers. That something is a function.

In the parent there are two such boxes stacked: $f$ (the encoder, e.g. a ViT or ResNet) then $g$ (the small projection head). Composition just means "run one box, feed its output into the next": $z=g(f(x))$.

3. The dot product: measuring alignment

Now we have two dots. We need a number that says how much they point the same way. That is the dot product.

The little $\top$ (read "transpose") is bookkeeping notation that flips a column list into a row so the multiplication lines up. For our purposes $u^{\top }v$ just means "dot product of $u$ and $v$."

4. Length of a vector, and cosine similarity

The raw dot product has a flaw: a longer arrow gives a bigger number even if it points the same way. We only care about direction, not how long the arrow is. So we divide out the lengths.

5. Exp and log: turning scores into probabilities

We now have a similarity score per pair. To train, we want to phrase things as a probability: "how likely is this the correct match?" Two functions convert scores↔probabilities: $\exp$ and $\log$.

6. Sum ($\sum$), softmax, and temperature $\tau$

Now stack the tools. Take similarities, run each through $\exp$, and divide each by the total. Every result is positive and they all add to $1$ — that's a probability distribution. This recipe is softmax.

Putting it together, the parent's InfoNCE loss for anchor $z$ with positive $z^{+}$ and negatives $z_i^{-}$ is just "negative log of the softmax probability that the positive is the right answer": $\mathcal{L}=-\log \frac{\exp (\text{sim}(z,z^{+})/\tau )}{\exp (\text{sim}(z,z^{+})/\tau )+{\sum }_i\exp (\text{sim}(z,z_i^{-})/\tau )}$ Read left to right with our toolbox: measure alignment ($\text{sim}$) → sharpen ($/\tau$) → make positive ($\exp$) → convert to a share (the fraction) → make it a pain meter ($-\log$). Nothing mysterious remains.

7. Two more words the parent leans on

These connect outward to Transfer learning and fine-tuning (reuse the learned map), Zero-shot and few-shot learning (CLIP classifies unseen categories), Multi-modal learning (images + text share one map), and Attention mechanisms and transformers (the encoders themselves). Training stability tricks live in Batch normalization and training techniques.

Prerequisite map

Equipment checklist

Test yourself — answer before revealing.