Text-to-image conditioning (CLIP)

4.5.16 · AI-ML › Generative Models

CLIP Sab Kuch Kyun Badal Deta Hai

CLIP se pehle, text-to-image models ko chahiye tha:

1. Images ke liye exact captions ke saath supervised datasets
2. Fixed vocabulary (agar training mein "steampunk octopus" nahi tha, toh generate nahi ho sakta)
3. Classification, generation, etc. ke liye alag alag models

CLIP zero-shot transfer enable karta hai kyunki yeh concepts seekhta hai, labels nahi. Ek baar train hone ke baad, yeh diffusion models ko ANY text prompt se images generate karne mein guide kar sakta hai, chahe woh combination pehle kabhi dekha hi na ho.

Dono embeddings ek hi $d$-dimensional space mein rehte hain (typically $d=512$ ya $d=768$). Key baat: contrastive loss matching pairs ko similar aur non-matching pairs ko dissimilar banata hai.

Contrastive Loss Ko First Principles Se Derive Karna

Goal: $N$ image-text pairs $(x_i,t_i)$ ke ek batch ko deke, sahi pairs ke liye similarity maximize karo, galat ones ke liye minimize karo.

Step 1: Similarities Compute Karo

Har image $i$ aur text $j$ ke liye, cosine similarity compute karo:

$s_{ij}=\frac{{\mathbf{v}}_I^{(i)}\cdot {\mathbf{v}}_T^{(j)}}{\Vert {\mathbf{v}}_I^{(i)}\Vert \Vert {\mathbf{v}}_T^{(j)}\Vert }=\text{cos}({\mathbf{v}}_I^{(i)},{\mathbf{v}}_T^{(j)})$

Cosine kyun? Hume direction (semantic meaning) ki parwah hai, magnitude ki nahi. Normalized embeddings ka matlab hai $s_{ij}\in [-1,1]$.

Step 2: Temperature Scaling

Sharpness control karne ke liye temperature $\tau$ apply karo:

${\text{logits}}_{ij}=\frac{s_{ij}}{\tau }$

Temperature kyun? Chhota $\tau$ softmax ko sharper banata hai (zyada confident), bada $\tau$ use smoother banata hai. Typically $\tau \approx 0.07$. Yeh ek learned parameter hai.

Step 3: Symmetric Cross-Entropy

Image $i$ ke liye, probability ki text $j$ uska match hai:

$p(j|i)=\frac{\exp ({\text{logits}}_{ij})}{{\sum }_{k=1}^N\exp ({\text{logits}}_{ik})}$

Image-to-text loss (image $i$ ko apna text $i$ predict karna chahiye):

${\mathcal{L}}_{I\to T}=-\frac{1}{N}{\sum }_{i=1}^N\log p(i|i)=-\frac{1}{N}{\sum }_{i=1}^N\log \frac{\exp (s_{ii}/\tau )}{{\sum }_{j=1}^N\exp (s_{ij}/\tau )}$

Yeh form kyun? Yeh predicted distribution aur ek one-hot target (image $i$ sirf text $i$ se match karta hai) ke beech cross-entropy hai.

Symmetrically, text-to-image loss:

${\mathcal{L}}_{T\to I}=-\frac{1}{N}{\sum }_{j=1}^N\log \frac{\exp (s_j/\tau )}{{\sum }_{i=1}^N\exp (s_{ij}/\tau )}$

Final CLIP loss (symmetric):

${\mathcal{L}}_{\text{CLIP}}=\frac{1}{2}({\mathcal{L}}_{I\to T}+{\mathcal{L}}_{T\to I})$

Symmetric kyun? Dono directions mein training karne se mode collapse rokta hai aur ensure karta hai ki dono encoders rich representations seekhein.

${\widetilde{ϵ}}_{\theta }(z_t,t)={ϵ}_{\theta }(z_t,t,\varnothing )+w\cdot [{ϵ}_{\theta }(z_t,t,c)-{ϵ}_{\theta }(z_t,t,\varnothing )]$

Jahan:

• $c$ = CLIP text embedding ${\mathbf{v}}_T$
• $w$ = guidance scale (typically 7-15)
• $\varnothing$ = empty/null conditioning

Yeh kaam kyun karta hai? Difference $[{ϵ}_{\theta }(z_t,t,c)-{ϵ}_{\theta }(z_t,t,\varnothing )]$ text condition ki taraf point karta hai. $w$ se scale karna is direction ko amplify karta hai, generation ko prompt ke zyada faithful banata hai.

Guidance Term Derive Karna

Conditional score ke liye Bayes' rule se shuru karo:

${\nabla }_{z_t}\log p(z_t|c)={\nabla }_{z_t}\log p(z_t)+{\nabla }_{z_t}\log p(c|z_t)$

Classifier gradient ${\nabla }_{z_t}\log p(c|z_t)$ condition ki taraf steer karta hai. Lekin hamare paas alag classifier nahi hai!

Classifier-free trick: Usi model ko train karo conditioning randomly drop karke (10-20% time). Phir:

${\nabla }_{z_t}\log p(c|z_t)\approx {ϵ}_{\theta }(z_t,t,c)-{ϵ}_{\theta }(z_t,t,\varnothing )$

Kyun? Score ${ϵ}_{\theta }(z_t,t,c)$ ne conditional distribution seekhi, aur ${ϵ}_{\theta }(z_t,t,\varnothing )$ ne marginal seekhi. Unka difference approximate karta hai ki condition distribution ko kitna shift karta hai.

Step 1: Encode karo

• Images → embeddings: ${\mathbf{v}}_I^{(1)},{\mathbf{v}}_I^{(2)},{\mathbf{v}}_I^{(3)}$ (har ek ${\mathbb{R}}^{512}$ mein)
• Texts → embeddings: ${\mathbf{v}}_T^{(1)},{\mathbf{v}}_T^{(2)},{\mathbf{v}}_T^{(3)}$

Step 2: Similarity matrix $S$ compute karo (3×3):

s_{11} & s_{12} & s_{13} \\ s_{21} & s_{22} & s_{23} \\ s_{31} & s_{32} & s_{33} \end{bmatrix}$$ Jahan $s_{ij} = \frac{\mathbf{v}_I^{(i)} \cdot \mathbf{v}_T^{(j)}}{\|\mathbf{v}_I^{(i)}\| \|\mathbf{v}_T^{(j)}\|}$ **Sabhi pairs compute kyun karo?** Contrastive learning ko negatives chahiye! Pair (1,1) positive hai, lekin (1,2) aur (1,3) negatives hain. **Step 3**: Temperature aur softmax apply karo $$\text{logits} = S / \tau = \begin{bmatrix} s_{11}/0.07 & s_{12}/0.07 & s_{13}/0.07 \\ \vdots & \vdots & \vdots \end{bmatrix}$$ Image 1 ke liye, probabilities: $$p(j|1) = \frac{\exp(s_{1j}/0.07)}{\exp(s_{11}/0.07) + \exp(s_{12}/0.07) + \exp(s_{13}/0.07)}$$ **Yeh kyun matter karta hai?** Hum chahte hain $p(1|1)$ (dog image → "a dog") 1 ke karib ho, matlab $s_{11} \gg s_{12}, s_{13}$. **Step 4**: Image 1 ke liye loss $$\mathcal{L}_1 = -\log p(1|1) = -\log \frac{\exp(s_{11}/0.07)}{\sum_j \exp(s_{1j}/0.07)}$$ Sabhi 3 images pe average karo, phir texts → images ke liye bhi same karo. Final loss average hai. **Outcome**: Gradients $\mathbf{v}_I^{(1)}$ aur $\mathbf{v}_T^{(1)}$ ko paas laate hain ($s_{11}$ badhata hai), $\mathbf{v}_I^{(1)}$ ko $\mathbf{v}_T^{(2)}, \mathbf{v}_T^{(3)}$ se door dhakelte hain ($s_{12}, s_{13}$ ghatate hain). > [!example] Example 2: Diffusion Guidance ke liye CLIP Use Karna > **Task**: "a futuristic city at sunset" generate karo guidance scale $w=7.5$ ke saath. **Step 1**: Prompt encode karo $$c = f_T(\text{"a futuristic city at sunset"}) \in \mathbb{R}^{512}$$ **Step 2**: Har diffusion step $t$ pe, model noise predict karta hai: - Conditional: $\epsilon_\theta(z_t, t, c)$ (prompt ke baare mein jaanta hai) - Unconditional: $\epsilon_\theta(z_t, t, \emptyset)$ (koi prompt nahi, sirf general images seekhta hai) **Dono kyun compute karo?** Difference jaanne ke liye ki prompt hume kis direction mein steer karta hai. **Step 3**: Guided noise compute karo $$\tilde{\epsilon} = \epsilon_\theta(z_t, t, \emptyset) + 7.5 \cdot [\epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \emptyset)]$$ **Yeh step kyun?** Term $[\epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \emptyset)]$ ek vector hai jo "prompt ke zyada jaisi" direction mein point karta hai. 7.5 se multiply karna ise amplify karta hai, image ko prompt se zyada strongly match karata hai. **Step 4**: $\tilde{\epsilon}$ use karke denoise karo $$z_{t-1} = \frac{1}{\sqrt{\alpha_t}}(z_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \tilde{\epsilon}) + \sigma_t \mathbf{\epsilon}$$ **Outcome**: $w=7.5$ ke saath, image strongly "futuristic city at sunset" jaisi lagti hai. $w=1$ ke saath, yeh weaker hai (zyada diverse lekin kam accurate). $w=0$ ke saath, yeh sirf random images hain. > [!example] Example 3: Zero-Shot Classification > **Task**: CLIP use karke ek kutte ki image ko "dog", "cat", ya "bird" classify karo (kabhi classifier ke roop mein train nahi kiya gaya!). **Step 1**: Image encode karo $$\mathbf{v}_I = f_I(\text{dog image}) \in \mathbb{R}^{512}$$ **Step 2**: Candidate labels ko prompts ke roop mein encode karo - $\mathbf{v}_{T,\text{dog}} = f_T(\text{"a photo of a dog"})$ - $\mathbf{v}_{T,\text{cat}} = f_T(\text{"a photo of a cat"})$ - $\mathbf{v}_{T,\text{bird}} = f_T(\text{"a photo of a bird"})$ **"a photo of" kyun?** Yeh CLIP ki training distribution se bare labels ke comparison mein zyada match karta hai. **Step 3**: Similarities compute karo $$s_{\text{dog}} = \mathbf{v}_I \cdot \mathbf{v}_{T,\text{dog}}, \quad s_{\text{cat}} = \mathbf{v}_I \cdot \mathbf{v}_{T,\text{cat}}, \quad s_{\text{bird}} = \mathbf{v}_I \cdot \mathbf{v}_{T,\text{bird}}$$ **Step 4**: Probabilities ke liye softmax $$p(\text{dog}) = \frac{\exp(s_{\text{dog}}/\tau)}{\exp(s_{\text{dog}}/\tau) + \exp(s_{\text{cat}}/\tau) + \exp(s_{\text{bird}}/\tau)}$$ **Yeh kaam kyun karta hai?** CLIP ne seekha ki dog images embedding space mein "a photo of a dog" ke paas hoti hain. Koi explicit classification training nahi chahiye—yeh contrastive learning se emerge hota hai! **Outcome**: $p(\text{dog}) \approx 0.92$, $p(\text{cat}) \approx 0.05$, $p(\text{bird}) \approx 0.03$. Prediction: dog. > [!mistake] Galti 1: Yeh Sochna Ki CLIP Pixel-Level Understanding Karta Hai > **Galat Idea**: "CLIP spatial relationships samajh sakta hai jaise 'the cat is on the left of the dog'." **Yeh Sahi Kyun Lagta Hai**: CLIP complex prompts handle karta hai aur intelligent lagta hai, isliye use detailed layouts samajhne chahiye. **Reality**: CLIP embeddings ==global pooled representations== hain. Image encoder (usually ek Vision Transformer ya ResNet) pooling ke baad ek image ke liye ek single vector output karta hai. "Left of" ya "behind" jaisi spatial information zyada tar kho jaati hai. CLIP **object presence aur style** mein accha hai, lekin struggle karta hai: - Precise spatial relationships ke saath - Objects count karna ("three dogs") - Fine-grained attributes ("the dog with a red collar") **Yeh Kyun Hota Hai**: Contrastive loss sirf global level pe image-text similarity ki parwah karta hai. Image ke regions ko text ke parts ke saath align karne ka koi mechanism nahi hai (unlike, say, region-based captioning models). **Fix**: Spatial control ke liye, additional mechanisms use karo: 1. **Cross-attention** (jaise DALL-E 2 ya Stable Diffusion mein): Diffusion model ke cross-attention layers ko specific text tokens pe attend karne do har spatial location pe. 2. **Layout conditioning**: Text ke saath bounding boxes ya segmentation maps provide karo. 3. **Composite prompts**: Task ko multiple generations mein toddo aur unhe composite karo. > [!mistake] Galti 2: CLIP Similarity Ko Naively Perceptual Loss Ki Tarah Use Karna > **Galat Idea**: "High CLIP similarity = prompt se perfect image match, toh main ise directly maximize karunga." **Yeh Sahi Kyun Lagta Hai**: CLIP similarity exactly wahi hai jo hum chahte hain—text-image alignment—toh ise optimize karna best image dega. **Reality**: CLIP similarity ko directly optimize karna (e.g., $\mathbf{v}_I \cdot \mathbf{v}_T$ pe gradient ascent) ==adversarial examples== tak le jaata hai—aisi images jo similarity maximize karti hain lekin humans ko noise ya weird artifacts jaisi lagti hain. CLIP ==typographic attacks== (image mein text add karna) aur texture biases ke liye vulnerable hai. **Yeh Kyun Hota Hai**: CLIP contrastive loss se train hua tha, perfect perceptual metric ke roop mein nahi. Usne shortcuts seekhe: certain texture patterns, colors, ya image mein text bhi similarity artificially boost kar sakta hai. Yeh natural images ki full distribution model nahi karta. **Fix**: 1. **CLIP ko guidance ki tarah use karo, direct optimization ki tarah nahi**: Diffusion mein, CLIP score conditioning ke through denoising process guide karta hai, jo image manifold pe rehta hai. 2. **Doosre losses ke saath combine karo**: CLIP similarity ko perceptual losses (LPIPS), pixel losses, ya adversarial losses ke saath mix karo. 3. **Regularization**: Smoothness penalties add karo ya ek learned image prior ke andar raho (e.g., GAN latent space). **Text-to-image ke liye better approach**: CLIP embeddings ko ek generative model (diffusion, GAN, autoregressive) ke liye conditioning ki tarah use karo, direct optimization target ki tarah nahi. > [!recall]- Feynman Explanation (ek 12-saal ke bachche ke liye) > Socho tumhare paas LEGO pieces ka ek bada box hai (images) aur instruction manuals ka ek bada box hai (text descriptions). Tum ek robot ko sikhana chahte ho ki kaunsa manual kaunse LEGO model ke saath jaata hai, lekin tum har possible pair check nahi kar sakte—millions hain! CLIP ka trick kuch aisa hai: yeh ek LEGO model leta hai, use ek "fingerprint" mein squish karta hai (numbers ki ek list), aur manual ke saath bhi same karta hai. Agar woh match karte hain, toh unke fingerprints bahut similar hone chahiye (jaise twins). Agar match nahi karte, toh fingerprints bilkul alag hone chahiye (jaise tum aur koi stranger). Robot 100 models aur 100 manuals ek saath dekh ke seekhta hai. Har model ke liye, woh sahi manual chunne ki koshish karta hai. Agar galat chune, toh penalty milti hai aur adjust karta hai. Time ke saath, woh bahut accha ho jaata hai yeh jaanne mein ki "yeh dragon model 'build a dragon' manual se match karta hai." Ab yahan magic hai: ek baar millions of pairs seek lene ke baad, tum ise ek NAYA manual de sakte ho jo kehta hai "build a spaceship made of candy" (kuch jo usne pehle kabhi nahi dekha), aur woh scratch se sahi LEGO model CREATE kar sakta hai! Use har possible cheez memorize nahi karni padti—usne spaceship aur candy kaise dikhte hain ki IDEA seekhi. Isi tarah CLIP computers ko kisi bhi text description se images generate karne deta hai, chahe woh crazy ones hon jo pehle kisi ne socha bhi na ho. > [!mnemonic] CLIP = **C**ontrastive **L**anguage-**I**mage **P**re-training > **Memory hook**: CLIP images aur text ko ek paperclip ki tarah "clip" karta hai—matching pairs attached rehte hain, mismatches alag ho jaate hain. **Formula memory**: - **Symmetric loss** = Do-taraf ki road (image→text AUR text→image) - **Temperature** $\tau$ = Heat badhaao taaki decisions sharper ho jayein (chhota $\tau$ = confident, bada $\tau$ = uncertain) - **Guidance scale** $w$ = Volume knob ke liye ki text prompt kitna loud hai (zyada $w$ = image prompt ko chillata hai) --- ## Connections - [[4.5.14-Diffusion-models]] - CLIP text-to-image diffusion ke liye conditioning mechanism provide karta hai - [[4.5.15-Classifier-free-guidance]] - CLIP joint embeddings seekh ke classifier-free guidance enable karta hai - [[4.3.8-Contrastive-learning]] - CLIP scale pe contrastive learning ka premier example hai - [[4.2.12-Transfer-learning]] - CLIP ki zero-shot capabilities ultimate transfer learning hai - [[4.4.7-Attention-mechanisms]] - CLIP mein Vision Transformers image encoding ke liye self-attention use karte hain - [[4.5.17-Multimodal-learning]] - CLIP ek foundational multimodal model hai --- #flashcards/ai-ml CLIP ke peeche core idea kya hai? :: CLIP ek shared embedding space seekhta hai jahan matching text-image pairs paas paas hote hain aur mismatched pairs door, enabling zero-shot transfer aur text-to-image conditioning bina task-specific training ke. CLIP Euclidean distance ki jagah cosine similarity kyun use karta hai? ::: Cosine similarity direction (semantic meaning) measure karta hai na ki magnitude, aur normalized embeddings ensure karte hain ki similarities [-1, 1] mein bounded hoon, inhe softmax mein kaam karna aasaan banata hai. CLIP contrastive loss ko image-to-text ke liye derive karo. ::: Image $i$ ke liye, sabhi texts $j$ ke saath similarity $s_{ij}$ compute karo, temperature scaling $s_{ij}/\tau$ apply karo, phir cross-entropy use karo: $\mathcal{L}_{I \to T} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(s_{ii}/\tau)}{\sum_j \exp(s_{ij}/\tau)}$. Symmetric loss mein text-to-image bhi shamil hai. CLIP mein temperature $\tau$ ka kya role hai? :: Temperature $\tau$ softmax distribution ki sharpness control karta hai. Chhota $\tau$ (e.g., 0.07) model ko zyada confident banata hai (sharp peaks), jabki bada $\tau$ distribution ko smooth karta hai. Yeh ek learned parameter hai jo hard aur soft assignments ke beech balance karta hai. CLIP loss symmetric kyun hai? ::: Image-to-text aur text-to-image dono directions mein training mode collapse rokti hai aur ensure karti hai ki dono encoders rich, balanced representations seekhein. Yeh contrastive learning ke liye effective batch size bhi double kar deta hai. Classifier-free guidance CLIP ke saath kaise kaam karta hai? ::: Model ko conditioning randomly drop karke train kiya jaata hai (10-20% time). Inference pe, guided noise hai $\tilde{\epsilon} = \epsilon_\theta(z_t, \emptyset) + w \cdot [\epsilon_\theta(z_t, c) - \epsilon_\theta(z_t, \emptyset)]$, jahan difference condition ki taraf classifier gradient approximate karta hai. Guidance scale $w$ kya control karta hai? ::: Guidance scale $w$ control karta hai ki generated image text prompt ke kitni strongly adheres karti hai. Zyada $w$ (7-15) prompt ke zyada faithful images produce karta hai lekin kam diverse; kam $w$ zyada variety deta hai lekin weaker correspondence. CLIP zero-shot classification kyun kar sakta hai? ::: CLIP ek shared embedding space mein semantic concepts seekhta hai. Ek image classify karne ke liye, use aur candidate label texts encode karo, similarities compute karo, aur sabse high wala lo. Koi explicit classification training nahi chahiye—yeh contrastive learning se emerge hota hai. Spatial reasoning ke liye CLIP ki main limitations kya hain? ::: CLIP global pooled representations use karta hai, spatial information kho deta hai. Yeh precise spatial relationships ("left of"), objects count karne, aur fine-grained attributes ke saath struggle karta hai kyunki contrastive loss sirf global image-text level pe align karta hai. Image generation ke liye CLIP similarity directly optimize kyun nahi karni chahiye? ::: Direct optimization adversarial examples tak le jaati hai—aisi images jo similarity maximize karti hain lekin noise ya artifacts contain karti hain. CLIP mein shortcuts hain (textures, colors, typographic attacks) aur ise perceptual metric ki tarah train nahi kiya gaya. Iske bajaye ise generative models ke liye guidance ki tarah use karo. CLIP text-to-image diffusion models ko kaise enable karta hai? ::: CLIP text embeddings diffusion model ke denoising process ko cross-attention ya guidance ke through condition karti hain. Model seekhta hai denoise karna un images ki taraf jo prompt ke saath high CLIP similarity rakhti hain, bina har concept ke liye paired training data ki zaroorat ke. CLIP mnemonic mein "clip" kya hai? ::: CLIP images aur text ko ek paperclip ki tarah "clips" karta hai—matching pairs attached rehte hain (high similarity), jabki mismatches alag ho jaate hain (low similarity). Contrastive loss is attachment ko enforce karta hai. ## 🖼️ Concept Map ```mermaid flowchart TD IMG[Image x] -->|f_I encoder| VI[Image embedding v_I] TXT[Text prompt t] -->|f_T encoder| VT[Text embedding v_T] VI -->|shared d-dim space| SIM[Cosine similarity s_ij] VT -->|shared d-dim space| SIM SIM -->|scale by temperature tau| LOG[Logits] LOG -->|softmax cross-entropy| LIT[Image-to-text loss] LOG -->|softmax cross-entropy| LTI[Text-to-image loss] LIT -->|averaged| CL[CLIP contrastive loss] LTI -->|averaged| CL CL -->|trains encoders| SHARED[Aligned embedding space] SHARED -->|enables| ZS[Zero-shot text-to-image guidance] ZS -->|steers| DIFF[Diffusion generation] ```