Multimodal models (vision-language)

6.5.5 · AI-ML › Research Frontiers & Practice

What Problem Do They Solve?

Traditional AI mein alag alag models hote the:

• Vision models: Images classify karo → "this is a cat" (sirf label)
• Language models: Text generate karo → "The cat sat on the mat" (sirf text)

Lekin asli intelligence ke liye cross-modal reasoning chahiye:

• Image captioning: Jo dikhe usse describe karo
• Visual question answering: "What color is the car?" — iske liye dono modalities chahiye
• Text-to-image search: "sunset over mountains" match karne wali images dhundo
• Zero-shot classification: Aisi objects ko recognize karo jinhe explicitly train nahi kiya, text descriptions se matching karke

Core challenge: Do bilkul alag data types (pixel grids vs. discrete tokens) ko ek common representation mein align kaise karein?

Architecture: Building the Bridge

Step-by-Step Construction

WHY separate encoders? Images continuous 2D grids hain; text discrete 1D sequences hain. Dono ko alag inductive biases chahiye.

1. Vision Encoder (Image → Vectors)

WHAT: Raw pixels ko feature vectors mein convert karta hai.

HOW: Modern approaches Vision Transformers (ViT) ya CNN backbones use karte hain.

ViT ke liye:

1. Image ko patches mein split karo (jaise 16×16 pixels)
2. Har patch ko flatten karke ek vector banao
3. Positional embeddings add karo (taaki model spatial layout jaane)
4. Transformer layers se pass karo

$\text{Image }I\in {\mathbb{R}}^{H\times W\times 3}\to \text{Patches }\{p_1,p_2,\dots ,p_N\}$

$\text{Patch embedding: }{z}_i=\text{Linear}(\text{Flatten}(p_i))+{\text{PE}}_i$

jahan PE = positional encoding, $N=\frac{HW}{P^2}$ patch size $P$ ke liye.

WHY transformers for vision? Ye long-range dependencies capture karte hain (ek dog ki tail uske head se image ke across related hoti hai) jo CNNs ke local receptive fields se behtar hai.

2. Text Encoder (Text → Vectors)

WHAT: Tokenized text ko feature vectors mein convert karta hai.

HOW: Typically ek Transformer encoder (jaise BERT):

1. Text ko subwords mein tokenize karo
2. Token + positional embeddings add karo
3. Transformer layers se pass karo
4. Final representation extract karo (aksar [CLS] token ya mean pooling)

$\text{Text }T\to \text{Tokens }\{t_1,t_2,\dots ,t_M\}$

$h_j=\text{Transformer}(\text{Embed}(t_j)+{\text{PE}}_j)$

3. Joint Embedding Space (The Bridge)

WHY: Hume image vectors $v_{\text{img}}\in {\mathbb{R}}^d$ aur text vectors $v_{\text{text}}\in {\mathbb{R}}^d$ ko same dimensional space mein chahiye taaki unhe compare kar sakein.

HOW: Dono modalities ko dimension $d$ pe learned linear layers se project karo:

$v_{\text{img}}=W_v\cdot h_{\text{img}}^{\text{final}}$ $v_{\text{text}}=W_t\cdot h_{\text{text}}^{\text{final}}$

Phir cosine similarity se similarity measure karo:

WHY cosine? Ye direction alignment measure karta hai magnitude se independent hokar — hume parwah hai ki concepts embedding space mein same direction mein point kar rahe hain ya nahi, unki absolute size ki nahi.

Values range $[-1,1]$ mein:

• +1 = perfect alignment (same concept)
• 0 = orthogonal (unrelated)
• -1 = opposite

Training Objective: Contrastive Learning

Image $i$ ke liye matching text $t_i$ ke saath:

${\mathcal{L}}_i=-\log \frac{\exp (\text{sim}(v_i,v_{t_i})/\tau )}{{\sum }_{j=1}^B\exp (\text{sim}(v_i,v_{t_j})/\tau )}$

jahan $\tau$ ek learned temperature parameter hai jo sharpness control karta hai.

Derivation from first principles:

1. Intuition: Hum softmax classification kar rahe hain jahan correct text $B$ candidates mein se "label" hai.

2. Why exponential? $\exp ()$ similarities ko positive probabilities mein convert karta hai.

3. Why divide by sum? Normalization taaki ek proper probability distribution bane.

4. Temperature $\tau$:

   • Chhota $\tau$ → sharper distribution (model ko bahut confident hona padega)
   • Bada $\tau$ → softer distribution (zyada uncertainty allowed) Typically $\tau \approx 0.07$. Yeh seekha jaata hai kyunki optimal "confidence" dataset ke hisaab se vary karti hai.

5. Full loss: Dono directions ka average (image→text aur text→image):

$\mathcal{L}=\frac{1}{2}({\mathcal{L}}_{\text{img→txt}}+{\mathcal{L}}_{\text{txt→img}})$

WHY symmetric? Agar "cat photo" → "cat text" close hona chahiye, toh "cat text" → "cat photo" bhi close hona chahiye. Yeh degenerate solutions rokta hai.

Setup: 4 (image, caption) pairs ka batch:

1. (dog photo, "a golden retriever")
2. (cat photo, "a tabby cat")
3. (car photo, "a red sedan")
4. (tree photo, "an oak tree")

Step 1: Saari images aur saare texts encode karo → total 8 vectors

Step 2: Similarity matrix compute karo (4×4):

         "dog"  "cat"  "car"  "tree"
dog_img   0.9   0.1    0.0    0.1
cat_img   0.1   0.8    0.0    0.2
car_img   0.0   0.1    0.9    0.0
tree_img  0.1   0.2    0.0    0.7

Step 3: Dog image ke liye, uski row pe softmax apply karo:

$P(\text{correct})=\frac{\exp (0.9/0.07)}{\exp (0.9/0.07)+\exp (0.1/0.07)+\exp (0.0/0.07)+\exp (0.1/0.07)}$

$=\frac{\exp (12.86)}{\exp (12.86)+\exp (1.43)+\exp (0)+\exp (1.43)}\approx 0.9999$

Loss = $-\log (0.9999)\approx 0.0001$ (bahut kam, model confident hai!)

Why this step? Exponentials 0.9 aur 0.1 similarity ke beech ke difference ko heavily amplify karte hain, jo model ko strong preferences commit karne pe majboor karta hai.

Step 4: Backpropagate karo taaki:

• dog_img aur "dog" text aur paas aayein (0.9 → 1.0 increase)
• dog_img baaki texts se door ho (0.1, 0.0 → aur negative)

Task: Kisi classifier ko train kiye bina ek image ko ["dog", "cat", "airplane"] mein classify karo.

How:

1. Image encode karo → $v_{\text{img}}$
2. Har class name ko text ke roop mein encode karo → $v_{\text{dog}},v_{\text{cat}},v_{\text{airplane}}$
3. Similarities compute karo: $[\text{sim}(v_{\text{img}},v_{\text{dog}}),\text{sim}(v_{\text{img}},v_{\text{cat}}),\text{sim}(v_{\text{img}},v_{\text{airplane}})]$
4. Highest similarity wali class predict karo

Why this works: Agar training ne "dog photos" ko "dog text" ke saath align kiya, toh test time pe ek unseen dog photo naturally "cat" text se zyada "dog" text ke paas hogi embedding space mein — semantic transfer.

Example calculation:

• Image: golden retriever photo
• Similarities: [0.85, 0.23, 0.05]
• Prediction: "dog" (highest at 0.85)

Why is airplane so low? Embedding space ne seekha ki dogs (furry, four legs, biological) airplanes (metallic, wings, mechanical) se semantically bahut door hain.

Key Architectural Variants

1. CLIP (Contrastive Language-Image Pre-training)

Architecture:

• Vision: ResNet ya ViT
• Text: Transformer encoder
• Joint space: 512-dim (ya 768-dim)
• Training: 400M (image, text) pairs internet se

Key innovation: Contrastive loss at scale — massive diverse data pe training se, extremely robust embeddings seekhta hai.

Use cases: Zero-shot classification, image search, content moderation

2. ALIGN (A Large-scale ImaGe and Noisy text)

Difference from CLIP: Noisy alt-text pe train kiya gaya (1.8B pairs) bina curation ke.

WHY noisy data works: Contrastive loss label noise ke liye robust hai — jab tak correct pairs thode bhi zyada similar hain wrong pairs se, signal billions of examples pe accumulate hoti hai.

3. Flamingo (DeepMind)

Architecture:

• Frozen vision encoder (CLIP se)
• Frozen LM (jaise Chinchilla 70B)
• Variable-length image features compress karne ke liye Perceiver Resampler
• LM mein insert ki gayi Cross-attention layers jo image features ko attend karti hain

Why this design? Pre-trained unimodal models (vision + language) ka faayda uthata hai aur sirf chhote "adapter" layers train karta hai — parameter efficient.

Use case: Few-shot visual question answering (2 examples dikhao, phir naye image ke baare mein jawab do)

4. BLIP-2 (Bootstrapping Language-Image Pre-training)

Architecture:

• Frozen image encoder
• Frozen LLM
• Querying Transformer (Q-Former) jo text generation ke liye relevant visual features extract karna seekhta hai

Training stages:

1. Vision-language representation learning: Contrastive + matching + captioning losses
2. Vision-to-language generative learning: Q-Former ke zariye frozen LM se connect karo

Why freeze? Scratch se training expensive hai; freezing se powerful pre-trained models ka faayda uthaya ja sakta hai.

Mathematical Deep Dive: Why Contrastive Loss Works

InfoNCE loss mutual information $I(v_{\text{img}};v_{\text{text}})$ ka ek lower bound hai.

Derivation:

Mutual information measure karta hai "image jaanne se text ke baare mein uncertainty kitni kam hoti hai":

$I(X;Y)=H(Y)-H(Y|X)$

jahan $H$ entropy hai.

$B$ negatives wala contrastive loss approximate karta hai:

$I(v_{\text{img}};v_{\text{text}})\ge \log (B)-{\mathcal{L}}_{\text{InfoNCE}}$

Why?

1. $B$ candidates ke saath, random guessing ki entropy $\log (B)$ hai
2. Hamara loss ise $\mathcal{L}$ tak reduce karta hai
3. Gap $\log (B)-\mathcal{L}$ woh information hai jo humne extract ki hai

Practical implication: Bade batch sizes $B$ → better training signal (contrast karne ke liye zyada hard negatives).

Typical CLIP $B=32,768$ use karta hai!

Common Mistakes

Why it feels right: Simple hai — sirf vectors stack karo aur jointly process karo.

Why it's wrong:

1. Dimension mismatch: Images 2048-dim features de sakti hain, text 768-dim — raw concatenation structure kho deta hai
2. No alignment: Concatenation model ko yeh nahi sikhata ki "dog photo" aur "dog word" same concept represent karte hain
3. Not interpretable: Retrieval nahi ho sakti (text ke liye matching images dhundna)

Fix: Projection layers use karo jo dono ko common dimension pe map karein, phir contrastive loss se semantics align karo.

Correct approach:

Image → CNN → Project to 512-dim → Normalize
Text → Transformer → Project to 512-dim → Normalize
Compute cosine similarity → Contrastive loss

Why it feels right: Ek kam hyperparameter tune karna padega.

Why it's wrong:

• $\tau =1$ distribution ko bahut soft bana deta hai — model confident hona nahi seekhta
• $\tau$ bahut chhota (jaise 0.01) gradient vanishing karta hai (softmax saturate ho jaata hai)

Example: Similarities [0.9, 0.3, 0.2] ke saath:

$\tau =1$: Softmax → [0.55, 0.24, 0.21] (sirf 55% correct pe!) $\tau =0.1$: Softmax → [0.996, 0.002] (99.6% correct pe)

Fix: $\tau$ ko ek parameter ki tarah seekho (initialized ~0.07) ya grid search karo. CLIP ise seekhta hai, $\log (1/0.07)\approx 2.66$ se shuru karke.

Why it feels right: Saare image patches aur saare text tokens ke beech deep cross-attention fine-grained alignments capture karegi.

Why it's partially wrong:

• Computational cost: $O(N_{\text{patches}}\times N_{\text{tokens}})$ attention expensive hai
• Overfitting: Chhote datasets pe complex fusion overfit karta hai
• Task dependent: Image-text retrieval ko sirf global similarity chahiye (simple embeddings kaam karti hain). Visual QA ko fine-grained reasoning chahiye (complex fusion help karta hai).

Fix: Architecture complexity ko task se match karo. Retrieval ke liye: dual encoders (CLIP). Generation ke liye: cross-attention (Flamingo, BLIP).

Implementation Details

Data Augmentation

Contrastive learning ke liye critical hai — shortcut learning rokta hai (model images ko low-level features jaise color se match na kare, semantics se kare).

Augmentations:

• Random crop & resize: Model ko objects ko alag scales pe recognize karne pe force karta hai
• Color jitter: Color distribution pe matching rokta hai
• Gaussian blur: Texture pe matching rokta hai
• Horizontal flip: Spatial invariance

WHY dono positive pairs ko identically augment karein? Taaki model invariant features seekhe — ek cat cat hi rehti hai chahe image ke left mein ho ya right mein.

Batch Size Scaling

Contrastive loss ko large batches chahiye (bahut saare negatives).

Standard training: Batch size 256-512 CLIP training: Batch size 32,768 (!)

How? Distributed training with gradient accumulation:

For each micro-batch of size 256:
  1. Forward pass → local loss
  2. Accumulate gradients (don't update yet)
After 128 micro-batches (= 32,768 total):
  3. Synchronize gradients across GPUs
  4. Update parameters

WHY this works: Contrastive loss negatives ke upar additive hota hai — gradients accumulate karna mathematically ek giant batch ke equivalent hai.

Socho tum picture cards aur word cards ke saath ek matching game khel rahe ho. Tumhe seekhna hai ki kaun si pictures kaun se words se match karti hain.

Computer yeh kaam ek "magic space" banake karta hai jahan wo dono pictures aur words rakhta hai. Agar picture aur word ka matlab same hai (jaise ek cat ki photo aur word "cat"), toh computer unhe is magic space mein paas paas rakhne ki koshish karta hai.

Yeh seekhne ke liye, wo ek game khelta hai: "Yahan ek dog ki picture hai. Kaun sa word best match karta hai: 'dog', 'cat', ya 'car'?" Agar wo 'dog' choose kare, toh gold star milta hai! Galat choose karne pe feedback milta hai ki dog picture ko 'dog' word ke aur paas le jao.

Lakhon baar alag alag pictures aur words ke saath yeh game khelne ke baad, computer sach mein achi tarah samajh jaata hai ki pictures aur words ka matlab same ho sakta hai — un cheezein ke liye bhi jo usne pehle kabhi nahi dekhi! Isliye yeh models naye photos describe kar sakte hain ya koi bhi description type karne pe matching pictures dhund sakte hain.

Active Recall

#flashcards/ai-ml

What is a multimodal vision-language model? :: Ek aisi neural architecture jisme images aur text ke liye alag encoders hote hain jo ek shared embedding space seekhte hain jahan cross-modal operations (retrieval, generation, reasoning) paired (image, text) data pe training se possible ho jaate hain.

Derive the contrastive loss (InfoNCE) for one image :: Given image $i$ with matching text $t_i$ in a batch of $B$ pairs: ${\mathcal{L}}_i=-\log \frac{\exp (\text{sim}(v_i,v_{t_i})/\tau )}{{\sum }_{j=1}^B\exp (\text{sim}(v_i,v_{t_j})/\tau )}$ where $\tau$ is temperature. Yeh softmax cross-entropy hai jo $B$ candidates mein correct text ko label maanke chalti hai.

Connections

• 6.2.03-Transformer-architecture - Vision Transformers self-attention ko image patches ke liye adapt karte hain
• 6.3.02-Self-supervised-learning - Contrastive learning massive scale pe self-supervision ka ek form hai
• 6.1.04-Word-embeddings - Joint embeddings word embeddings ko multiple modalities tak extend karti hain
• 6.4.01-Transfer-learning - Zero-shot classification fine-tuning ke bina extreme transfer hai
• 5.3.03-Cross-entropyloss - InfoNCE batch negatives pe softmax cross-entropy hai
• 6.2.05-Attention-mechanisms - Cross-attention fine-grained vision-language fusion enable karta hai
• 6.5.03-VisionTransformers - ViT modern multimodal models mein vision encoder ka kaam karta hai
• 6.5.06-Text-to-image-generation - Inverse task: aligned embeddings use karke text se images generate karo

Further Exploration

Research Papers:

• CLIP: Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021)
• ALIGN: Scaling Up Visual and Vision-Language Representation Learning (Jia et al., 2021)
• Flamingo: a Visual Language Model for Few-Shot Learning (Alayrac et al., 2022)
• BLIP-2: Bootstrapping Language-Image Pre-training (Li et al., 2023)

Key Insights:

1. Scale matters: CLIP ki success 400M pairs se aayi, novel architecture se nahi
2. Frozen pre-trained models: Modern approaches (Flamingo, BLIP-2) unimodal experts ko freeze karte hain aur sirf adapters train karte hain — parameter efficient hai aur existing capabilities ka faayda uthata hai
3. Emergent abilities: Aisi tasks pe zero-shot transfer jinka explicitly training nahi ki (OCR, counting, spatial reasoning)

Open Problems:

• Compositional reasoning: "a red cube on a blue sphere" vs "a blue cube on a red sphere"
• Counting: Current models "how many dogs?" se struggle karte hain
• Fine-grained alignment: Specific image regions ko text phrases se match karna (grounding)
• Temporal reasoning: Video-language tak extend karna (motion, events, causality)

Concept Map