Text-to-image conditioning (CLIP)

Why CLIP Changes Everything

Before CLIP, text-to-image models needed:

1. Supervised datasets with exact captions for images
2. Fixed vocabulary (can't generate "steampunk octopus" if it wasn't in training)
3. Separate models for classification, generation, etc.

CLIP enables zero-shot transfer because it learns concepts, not labels. Once trained, it can guide diffusion models to generate images from ANY text prompt, even novel combinations it never saw.

Both embedings live in the same $d$-dimensional space (typically $d=512$ or $d=768$). The key: contrastive loss makes matching pairs similar and non-matching pairs dissimilar.

Deriving the Contrastive Loss from First Principles

Goal: Given a batch of $N$ image-text pairs $(x_i, t_i)$, maximize similarity for correct pairs, minimize for incorrect ones.

Step 1: Compute Similarities

For each image $i$ and text $j$, compute cosine similarity:

$s_{ij} = \frac{\mathbf{v}_I^{(i)} \cdot \mathbf{v}_T^{(j)}}{\|\mathbf{v}_I^{(i)}\| \|\mathbf{v}_T^{(j)}\|} = \text{cos}(\mathbf{v}_I^{(i)}, \mathbf{v}_T^{(j)})$

Why cosine? We care about direction (semantic meaning), not magnitude. Normalized embedings mean $s_{ij} \in [-1, 1]$.

Step 2: Temperature Scaling

Apply temperature $\tau$ to control sharpness:

$\text{logits}_{ij} = \frac{s_{ij}}{\tau}$

Why temperature? Small $\tau$ makes the softmax sharper (more confident), large $\tau$ makes it smoother. Typically $\tau \approx 0.07$. This is a learned parameter.

Step 3: Symmetric Cross-Entropy

For image $i$, the probability that text $j$ is its match:

$p(j | i) = \frac{\exp(\text{logits}_{ij})}{\sum_{k=1}^N \exp(\text{logits}_{ik})}$

Image-to-text loss (image $i$ should predict its own text $i$):

$\mathcal{L}_{I \to T} = -\frac{1}{N} \sum_{i=1}^N \log p(i | i) = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^N \exp(s_{ij}/\tau)}$

Why this form? This is the cross-entropy between the predicted distribution and a one-hot target (image $i$ matches only text $i$).

Symmetrically, text-to-image loss:

$\mathcal{L}_{T \to I} = -\frac{1}{N} \sum_{j=1}^N \log \frac{\exp(s_{j}/\tau)}{\sum_{i=1}^N \exp(s_{ij}/\tau)}$

Final CLIP loss (symmetric):

$\mathcal{L}_{\text{CLIP}} = \frac{1}{2}(\mathcal{L}_{I \to T} + \mathcal{L}_{T \to I})$

Why symmetric? Training both directions prevents mode collapse and ensures both encoders learn rich representations.

$\tilde{\epsilon}_\theta(z_t, t) = \epsilon_\theta(z_t, t, \emptyset) + w \cdot [\epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \emptyset)]$

Where:

• $c$ = CLIP text embedding $\mathbf{v}_T$
• $w$ = guidance scale (typically 7-15)
• $\emptyset$ = empty/null conditioning

Why this works? The difference $[\epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \emptyset)]$ points toward the text condition. Scaling by $w$ amplifies this direction, making the generation more faithful to the prompt.

Deriving the Guidance Term

Start with Bayes' rule for conditional score:

$\nabla_{z_t} \log p(z_t | c) = \nabla_{z_t} \log p(z_t) + \nabla_{z_t} \log p(c | z_t)$

The classifier gradient $\nabla_{z_t} \log p(c | z_t)$ sters toward the condition. But we don't have a separate classifier!

Classifier-free trick: Train the same model with conditioning randomly dropped (10-20% of time). Then:

$\nabla_{z_t} \log p(c | z_t) \approx \epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \emptyset)$

Why? The score $\epsilon_\theta(z_t, t, c)$ learned the conditional distribution, and $\epsilon_\theta(z_t, t, \emptyset)$ learned the marginal. Their difference approximates how much the condition shifts the distribution.

Step 1: Encode

• Images → embeddings: $\mathbf{v}_I^{(1)}, \mathbf{v}_I^{(2)}, \mathbf{v}_I^{(3)}$ (each in $\mathbb{R}^{512}$)
• Texts → embeddings: $\mathbf{v}_T^{(1)}, \mathbf{v}_T^{(2)}, \mathbf{v}_T^{(3)}$

Step 2: Compute similarity matrix $S$ (3×3):

s_{11} & s_{12} & s_{13} \\ s_{21} & s_{22} & s_{23} \\ s_{31} & s_{32} & s_{33} \end{bmatrix}$$ Where $s_{ij} = \frac{\mathbf{v}_I^{(i)} \cdot \mathbf{v}_T^{(j)}}{\|\mathbf{v}_I^{(i)}\| \|\mathbf{v}_T^{(j)}\|}$ **Why compute all pairs?** Contrastive learning needs negatives! Pair (1,1) is positive, but (1,2) and (1,3) are negatives. **Step 3**: Apply temperature and softmax $$\text{logits} = S / \tau = \begin{bmatrix} s_{11}/0.07 & s_{12}/0.07 & s_{13}/0.07 \\ \vdots & \vdots & \vdots \end{bmatrix}$$ For image 1, probabilities: $$p(j|1) = \frac{\exp(s_{1j}/0.07)}{\exp(s_{11}/0.07) + \exp(s_{12}/0.07) + \exp(s_{13}/0.07)}$$ **Why this matters?** We want $p(1|1)$ (dog image → "a dog") to be close to 1, meaning $s_{11} \gg s_{12}, s_{13}$. **Step 4**: Loss for image 1 $$\mathcal{L}_1 = -\log p(1|1) = -\log \frac{\exp(s_{11}/0.07)}{\sum_j \exp(s_{1j}/0.07)}$$ Average over all 3 images, then do the same for texts → images. Final loss is the average. **Outcome**: Gradients push $\mathbf{v}_I^{(1)}$ and $\mathbf{v}_T^{(1)}$ closer (increase $s_{11}$), push $\mathbf{v}_I^{(1)}$ away from $\mathbf{v}_T^{(2)}, \mathbf{v}_T^{(3)}$ (decrease $s_{12}, s_{13}$). > [!example] Example 2: Using CLIP for Diffusion Guidance > **Task**: Generate "a futuristic city at sunset" with guidance scale $w=7.5$. **Step 1**: Encode the prompt $$c = f_T(\text{"a futuristic city at sunset"}) \in \mathbb{R}^{512}$$ **Step 2**: At each diffusion step $t$, the model predicts noise: - Conditional: $\epsilon_\theta(z_t, t, c)$ (knows about the prompt) - Unconditional: $\epsilon_\theta(z_t, t, \emptyset)$ (no prompt, just learns general images) **Why compute both?** We need the difference to know which direction the prompt sters us. **Step 3**: Compute guided noise $$\tilde{\epsilon} = \epsilon_\theta(z_t, t, \emptyset) + 7.5 \cdot [\epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \emptyset)]$$ **Why this step?** The term $[\epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \emptyset)]$ is a vector pointing toward "more like the prompt". Multiplying by 7.5 amplifies this, making the image match the prompt more strongly. **Step 4**: Denoise using $\tilde{\epsilon}$ $$z_{t-1} = \frac{1}{\sqrt{\alpha_t}}(z_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \tilde{\epsilon}) + \sigma_t \mathbf{\epsilon}$$ **Outcome**: With $w=7.5$, the image strongly resembles "futuristic city at sunset". With $w=1$, it's weaker (more diverse but less accurate). With $w=0$, it's just random images. > [!example] Example 3: Zero-Shot Classification > **Task**: Classify an image of a dog as "dog", "cat", or "bird" using CLIP (never trained as a classifier!). **Step 1**: Encode the image $$\mathbf{v}_I = f_I(\text{dog image}) \in \mathbb{R}^{512}$$ **Step 2**: Encode candidate labels as prompts - $\mathbf{v}_{T,\text{dog}} = f_T(\text{"a photo of a dog"})$ - $\mathbf{v}_{T,\text{cat}} = f_T(\text{"a photo of a cat"})$ - $\mathbf{v}_{T,\text{bird}} = f_T(\text{"a photo of a bird"})$ **Why "a photo of"?** This matches CLIP's training distribution better than bare labels. **Step 3**: Compute similarities $$s_{\text{dog}} = \mathbf{v}_I \cdot \mathbf{v}_{T,\text{dog}}, \quad s_{\text{cat}} = \mathbf{v}_I \cdot \mathbf{v}_{T,\text{cat}}, \quad s_{\text{bird}} = \mathbf{v}_I \cdot \mathbf{v}_{T,\text{bird}}$$ **Step 4**: Softmax to get probabilities $$p(\text{dog}) = \frac{\exp(s_{\text{dog}}/\tau)}{\exp(s_{\text{dog}}/\tau) + \exp(s_{\text{cat}}/\tau) + \exp(s_{\text{bird}}/\tau)}$$ **Why this works?** CLIP learned that dog images are close to "a photo of a dog" in embedding space. No explicit classification training needed—it's emergent from contrastive learning! **Outcome**: $p(\text{dog}) \approx 0.92$, $p(\text{cat}) \approx 0.05$, $p(\text{bird}) \approx 0.03$. Prediction: dog. > [!mistake] Mistake1: Thinking CLIP Does Pixel-Level Understanding > **Wrong Idea**: "CLIP can understand spatial relationships like'the cat is on the left of the dog'." **Why It Feels Right**: CLIP handles complex prompts and seems intelligent, so it should understand detailed layouts. **The Reality**: CLIP embedings are ==global pooled representations==. The image encoder (usually a Vision Transformer or ResNet) outputs a single vector per image after pooling. Spatial information like "left of" or "behind" is largely lost. CLIP is great at **object presence and style**, but struggles with: - Precise spatial relationships - Counting objects ("three dogs") - Fine-grained attributes ("the dog with a red collar") **Why This Happens**: The contrastive loss only cares about image-text similarity at the global level. There's no mechanism to align regions of the image with parts of the text (unlike, say, region-based captioning models). **The Fix**: For spatial control, use additional mechanisms: 1. **Cross-attention** (like in DALL-E 2 or Stable Diffusion): Let the diffusion model's cross-attention layers attend to specific text tokens at each spatial location. 2. **Layout conditioning**: Provide bounding boxes or segmentation maps alongside text. 3. **Composite prompts**: Break the task into multiple generations and composite them. > [!mistake] Mistake 2: Using CLIP Similarity as Perceptual Loss Naively > **Wrong Idea**: "High CLIP similarity = perfect image match to prompt, so I'll maximize it directly." **Why It Feels Right**: CLIP similarity is exactly what we want—text-image alignment—so optimizing it should give us the best image. **The Reality**: Direct optimization of CLIP similarity (e.g., gradient ascent on $\mathbf{v}_I \cdot \mathbf{v}_T$) leads to ==adversarial examples==—images that maximize similarity but look like noise or weird artifacts to humans. CLIP is vulnerable to ==typographic attacks== (adding text to the image) and texture biases. **Why This Happens**: CLIP was trained with contrastive loss, not as a perfect perceptual metric. It learned shortcuts: certain texture patterns, colors, or even text in the image can artificially boost similarity. It doesn't model the full distribution of natural images. **The Fix**: 1. **Use CLIP as guidance, not direct optimization**: In diffusion, CLIP guides the denoising process through score conditioning, which stays on the image manifold. 2. **Combine with other losses**: Mix CLIP similarity with perceptual losses (LPIPS), pixel losses, or adversarial losses. 3. **Regularization**: Add smoothness penalties or stay within a learned image prior (e.g., GAN latent space). **Better approach for text-to-image**: Use CLIP embedings as conditioning for a generative model (diffusion, GAN, autoregressive), not as a direct optimization target. > [!recall]- Feynman Explanation (for a12-year-old) > Imagine you have a huge box of LEGO pieces (images) and a huge box of instruction manuals (text descriptions). You want to teach a robot to know which manual goes with which LEGO model, but you can't check every possible pair—there are millions! CLIP's trick is like this: it takes a LEGO model, squishes it down into a "fingerprint" (a list of numbers), and does the same for the manual. If they're a match, their fingerprints should be really similar (like twins). If they don't match, the fingerprints should be totally different (like you and stranger). The robot learns by looking at 100 models and 100 manuals at once. For each model, it tries to pick the right manual. If it picks wrong, it gets a penalty and adjusts. Over time, it gets really good at knowing "this dragon model matches the'build a dragon' manual." Now here's the magic: once it's learned millions of pairs, you can give it a NEW manual that says "build a spaceship made of candy" (something it never saw before), and it can CREATE the right LEGO model from scratch! It doesn't need to memorize every possible thing—it learned the IDEA of what spaceships and candy look like. That's how CLIP lets computers generate images from any text description, even crazy ones nobody thought of before. > [!mnemonic] CLIP = **C**ontrastive **L**anguage-**I**mage **P**re-training > **Memory hook**: CLIP "clips" images and text together like a paperclip—matching pairs stay attached, mismatches fall apart. **Formula memory**: - **Symmetric loss** = Two-way street (image→text AND text→image) - **Temperature** $\tau$ = Turn up the heat to make decisions sharper (small $\tau$ = confident, large $\tau$ = uncertain) - **Guidance scale** $w$ = Volume knob for how loud the text prompt is (higher $w$ = image screams the prompt) --- ## Connections - [[4.5.14-Diffusion-models]] - CLIP provides the conditioning mechanism for text-to-image diffusion - [[4.5.15-Classifier-free-guidance]] - CLIP enables classifier-free guidance by learning joint embedings - [[4.3.8-Contrastive-learning]] - CLIP is the premier example of contrastive learning at scale - [[4.2.12-Transfer-learning]] - CLIP's zero-shot capabilities are the ultimate transfer learning - [[4.4.7-Attention-mechanisms]] - Vision Transformers in CLIP use self-attention for image encoding - [[4.5.17-Multimodal-learning]] - CLIP is a foundational multimodal model --- #flashcards/ai-ml What is the core idea behind CLIP? :: CLIP learns a shared embedding space where matching text-image pairs are close together and mismatched pairs are far apart, enabling zero-shot transfer and text-to-image conditioning without task-specific training. Why does CLIP use cosine similarity instead of Euclidean distance? ::: Cosine similarity measures direction (semantic meaning) rather than magnitude, and normalized embedings ensure similarities are bounded in [-1, 1], making them easier to work with in softmax. Derive the CLIP contrastive loss for image-to-text. ::: For image $i$, compute similarity $s_{ij}$ with all texts $j$, apply temperature scaling $s_{ij}/\tau$, then use cross-entropy: $\mathcal{L}_{I \to T} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(s_{ii}/\tau)}{\sum_j \exp(s_{ij}/\tau)}$. The symmetric loss also includes text-to-image. What is the role of temperature $\tau$ in CLIP? :: Temperature $\tau$ controls the sharpness of the softmax distribution. Small $\tau$ (e.g., 0.07) makes the model more confident (sharp peaks), while large $\tau$ smooths the distribution. It's a learned parameter that balances between hard and soft assignments. Why is the CLIP loss symmetric? ::: Training both image-to-text and text-to-image directions prevents mode collapse and ensures both encoders learn rich, balanced representations. It also doubles the effective batch size for contrastive learning. How does classifier-free guidance work with CLIP? ::: The model is trained with conditioning randomly dropped (10-20% of time). At inference, the guided noise is $\tilde{\epsilon} = \epsilon_\theta(z_t, \emptyset) + w \cdot [\epsilon_\theta(z_t, c) - \epsilon_\theta(z_t, \emptyset)]$, where the difference approximates the classifier gradient toward the condition. What does guidance scale $w$ control? ::: The guidance scale $w$ controls how strongly the generated image adheres to the text prompt. Higher $w$ (7-15) produces images more faithful to the prompt but less diverse; lower $w$ produces more variety but weaker correspondence. Why can CLIP do zero-shot classification? ::: CLIP learns semantic concepts in a shared embedding space. To classify an image, encode it and candidate label texts, compute similarities, and take the highest. No explicit classification training needed—it emerges from contrastive learning. What are CLIP's main limitations for spatial reasoning? ::: CLIP uses global pooled representations, losing spatial information. It struggles with precise spatial relationships ("left of"), counting objects, and fine-grained attributes because the contrastive loss only aligns at the global image-text level. Why shouldn't you directly optimize CLIP similarity for image generation? ::: Direct optimization leads to adversarial examples—images that maximize similarity but contain noise or artifacts. CLIP has shortcuts (textures, colors, typographic attacks) and wasn't trained as a perceptual metric. Use it as guidance for generative models instead. How does CLIP enable text-to-image diffusion models? ::: CLIP text embedings condition the diffusion model's denoising process through cross-attention or guidance. The model learns to denoise toward images that have high CLIP similarity with the prompt, without neding paired training data for every concept. What is the "clip" in the CLIP mnemonic? ::: CLIP "clips" images and text together like a paperclip—matching pairs stay attached (high similarity), while mismatches fall apart (low similarity). The contrastive loss enforces this attachment. ## 🖼️ Concept Map ```mermaid flowchart TD IMG[Image x] -->|f_I encoder| VI[Image embedding v_I] TXT[Text prompt t] -->|f_T encoder| VT[Text embedding v_T] VI -->|shared d-dim space| SIM[Cosine similarity s_ij] VT -->|shared d-dim space| SIM SIM -->|scale by temperature tau| LOG[Logits] LOG -->|softmax cross-entropy| LIT[Image-to-text loss] LOG -->|softmax cross-entropy| LTI[Text-to-image loss] LIT -->|averaged| CL[CLIP contrastive loss] LTI -->|averaged| CL CL -->|trains encoders| SHARED[Aligned embedding space] SHARED -->|enables| ZS[Zero-shot text-to-image guidance] ZS -->|steers| DIFF[Diffusion generation] ``` ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > ![[audio/4.5.16-Text-to-image-conditioning-(CLIP).mp3]] ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > Chalo, ise simple tareeke se samajhte hain. CLIP ka core idea yeh hai ki hum ek aisa **shared embedding space** banate hain jahaan text aur image dono ko numbers (vectors) mein convert kiya jaata hai. Matlab, "a photo of a dog" jaisa text aur ek actual kutte ki photo — dono ka vector agar semantically same hai, toh woh space mein ek dusre ke paas honge, aur jo match nahi karte woh door honge. Yeh kaam **contrastive loss** karta hai — jismein correct image-text pair ki cosine similarity maximize hoti hai aur galat pairs ki minimize. Yahaan temperature $\tau$ bas softmax ko sharp ya smooth banane ke liye hota hai, taaki model kitna confident ho yeh control ho. > > Ab yeh important kyun hai? Kyunki CLIP se pehle text-to-image models ko fixed labels aur exact captions waale supervised datasets chahiye the — agar "steampunk octopus" training mein nahi tha, toh generate hi nahi hota. CLIP **concepts** seekhta hai, labels nahi, isliye woh **zero-shot transfer** kar sakta hai — koi bhi naya prompt do, woh samajh leta hai. Yahi cheez usko diffusion models ke saath itna powerful banati hai, kyunki ab aap sirf natural language likhkar image generation ko steer kar sakte ho. > > Diffusion mein CLIP ka text embedding $c$ ban jaata hai aur **classifier-free guidance** ke through generation ko control karta hai. Idea yeh hai ki $[\epsilon_\theta(z_t,t,c) - \epsilon_\theta(z_t,t,\emptyset)]$ waala difference ek direction batata hai jo text condition ki taraf point karta hai — matlab "is prompt ki taraf jaao". Fir guidance scale $w$ (usually 7-15) se us direction ko amplify karte hain, jisse output aapke prompt ke zyada faithful banta hai. Toh short mein, CLIP hi woh bridge hai jo language aur images ko jodta hai, aur isi wajah se aaj hum text likhkar realistic images bana paate hain.