Multimodal models (vision-language)

What Problem Do They Solve?

Traditional AI had separate models:

• Vision models: Classify images → "this is a cat" (label only)
• Language models: Generate text → "The cat sat on the mat" (text only)

But real intelligence requires cross-modal reasoning:

• Image captioning: Describe what you see
• Visual question answering: "What color is the car?" needs both modalities
• Text-to-image search: Find images matching "sunset over mountains"
• Zero-shot classification: Recognize objects never explicitly trained on by matching with text descriptions

The core challenge: How do we align two completely different data types (pixel grids vs. discrete tokens) into a common representation?

Architecture: Building the Bridge

Step-by-Step Construction

WHY separate encoders? Images are continuous2D grids; text is discrete 1D sequences. They need different inductive biases.

1. Vision Encoder (Image → Vectors)

WHAT: Converts raw pixels to feature vectors.

HOW: Modern approaches use Vision Transformers (ViT) or CNN backbones.

For ViT:

1. Split image into patches (e.g., 16×16 pixels)
2. Flatten each patch into a vector
3. Add positional embedings (so model knows spatial layout)
4. Pass through transformer layers

$\text{Image } I \in \mathbb{R}^{H \times W \times 3} \rightarrow \text{Patches } \{p_1, p_2, \ldots, p_N\}$

$\text{Patch embedding: } z_i = \text{Linear}(\text{Flatten}(p_i)) + \text{PE}_i$

where PE = positional encoding, $N = \frac{HW}{P^2}$ for patch size $P$.

WHY transformers for vision? They capture long-range dependencies (a dog's tail relates to its head across the image) better than CNNs' local receptive fields.

2. Text Encoder (Text → Vectors)

WHAT: Converts tokenized text to feature vectors.

HOW: Typically a Transformer encoder (like BERT):

1. Tokenize text into subwords
2. Add token + positional embeddings
3. Pass through transformer layers
4. Extract final representation (often [CLS] token or mean pooling)

$\text{Text } T \rightarrow \text{Tokens } \{t_1, t_2, \ldots, t_M\}$

$h_j = \text{Transformer}(\text{Embed}(t_j) + \text{PE}_j)$

3. Joint Embedding Space (The Bridge)

WHY: We need image vectors $v_{\text{img}} \in \mathbb{R}^d$ and text vectors $v_{\text{text}} \in \mathbb{R}^d$ in the same dimensional space to compare them.

HOW: Project both modalities to dimension $d$ with learned linear layers:

$v_{\text{img}} = W_v \cdot h_{\text{img}}^{\text{final}}$ $v_{\text{text}} = W_t \cdot h_{\text{text}}^{\text{final}}$

Then measure similarity with cosine similarity:

WHY cosine? It measures direction alignment independent of magnitude—we care if concepts point the same way in embedding space, not their absolute size.

Values range $[-1, 1]$:

• +1 = perfect alignment (same concept)
• 0 = orthogonal (unrelated)
• -1 = opposite

Training Objective: Contrastive Learning

For image $i$ with matching text $t_i$:

$\mathcal{L}_i = -\log \frac{\exp(\text{sim}(v_i, v_{t_i}) / \tau)}{\sum_{j=1}^{B} \exp(\text{sim}(v_i, v_{t_j}) / \tau)}$

where $\tau$ is a learned temperature parameter controlling sharpness.

Derivation from first principles:

1. Intuition: We're doing softmax classification where the correct text is the "label" among $B$ candidates.

2. Why exponential? The $\exp()$ converts similarities to positive probabilities.

3. Why divide by sum? Normalization to make a proper probability distribution.

4. Temperature $\tau$:

   • Small $\tau$ → sharper distribution (model must be very confident)
   • Large $\tau$ → softer distribution (more uncertainty allowed) Typically $\tau \approx 0.07$. It's learned because optimal "confidence" varies by dataset.

5. Full loss: Average over both directions (image→text and text→image):

$\mathcal{L} = \frac{1}{2}(\mathcal{L}_{\text{img→txt}} + \mathcal{L}_{\text{txt→img}})$

WHY symmetric? If "cat photo" → "cat text" should be close, then "cat text" → "cat photo" should also be close. This prevents degenerate solutions.

Setup: Batch of 4 (image, caption) pairs:

1. (dog photo, "a golden retriever")
2. (cat photo, "a taby cat")
3. (car photo, "a red sedan")
4. (tree photo, "an oak tree")

Step 1: Encode all images and all texts →8 vectors total

Step 2: Compute similarity matrix (4×4):

         "dog"  "cat"  "car"  "tree"
dog_img   0.9   0.1    0.0    0.1
cat_img   0.1   0.8    0.0    0.2
car_img   0.0   0.1    0.9    0.0
tree_img  0.1   0.2    0.0    0.7

Step 3: For dog image, apply softmax over its row:

$P(\text{correct}) = \frac{\exp(0.9/0.07)}{\exp(0.9/0.07) + \exp(0.1/0.07) + \exp(0.0/0.07) + \exp(0.1/0.07)}$

$= \frac{\exp(12.86)}{\exp(12.86) + \exp(1.43) + \exp(0) + \exp(1.43)} \approx 0.9999$

Loss = $-\log(0.9999) \approx 0.0001$ (very low, model is confident!)

Why this step? The exponentials heavily amplify the difference between 0.9 and 0.1 similarity, making the model commit to strong preferences.

Step 4: Backpropagate to:

• Push dog_img and "dog" text closer (increase 0.9 → 1.0)
• Push dog_img away from other texts (decrease 0.1, 0.0 → more negative)

Task: Classify an image into ["dog", "cat", "airplane"] without training a classifier.

How:

1. Encode image → $v_{\text{img}}$
2. Encode each class name as text → $v_{\text{dog}}, v_{\text{cat}}, v_{\text{airplane}}$
3. Compute similarities: $[\text{sim}(v_{\text{img}}, v_{\text{dog}}), \text{sim}(v_{\text{img}}, v_{\text{cat}}), \text{sim}(v_{\text{img}}, v_{\text{airplane}})]$
4. Predict class with highest similarity

Why this works: If training aligned "dog photos" with "dog text", then at test time an unseen dog photo will naturally be closer to "dog" text than "cat" text in embedding space—semantic transfer.

Example calculation:

• Image: golden retriever photo
• Similarities: [0.85, 0.23, 0.05]
• Prediction: "dog" (highest at 0.85)

Why is airplane so low? The embedding space learned that dogs (fury, four legs, biological) are semantically distant from airplanes (metalic, wings, mechanical).

Key Architectural Variants

1. CLIP (Contrastive Language-Image Pre-training)

Architecture:

• Vision ResNet or ViT
• Text: Transformer encoder
• Joint space: 512-dim (or 768-dim)
• Training: 400M (image, text) pairs from internet

Key innovation: Contrastive loss at scale—by training on massive diverse data, learns extremely robust embedings.

Use cases: Zero-shot classification, image search, content moderation

2. ALIGN (A Large-scale ImaGe and Noisy text)

Difference from CLIP: Trained on noisy alt-text (1.8B pairs) without curation.

WHY noisy data works: Contrastive loss is robust to label noise—as long as correct pairs are slightly more similar than wrong pairs, the signal accumulates over billions of examples.

3. Flamingo (DepMind)

Architecture:

• Frozen vision encoder (from CLIP)
• Frozen LM (e.g., Chinchilla 70B)
• Perceiver Resampler to compress variable-length image features
• Cross-attention layers inserted into LM to attend to image features

Why this design? Leverages pre-trained unimodal models (vision + language) and only trains the small "adapter" layers—parameter efficient.

Use case: Few-shot visual question answering (show 2 examples, then answer about new image)

4. BLIP-2 (Bootstrapping Language-Image Pre-training)

Architecture:

• Frozen image encoder
• Frozen LLM
• Querying Transformer (Q-Former) learns to extract relevant visual features for text generation

Training stages:

1. Vision-language representation learning: Contrastive + matching + captioning losses
2. Vision-to-language generative learning: Connect to frozen LM via Q-Former

Why freeze? Training from scratch is expensive; freezing allows leveraging powerful pre-trained models.

Mathematical Deep Dive: Why Contrastive Loss Works

The InfoNCE loss is a lower bound on mutual information $I(v_{\text{img}}; v_{\text{text}})$.

Derivation:

Mutual information measures "how much knowing the image reduces uncertainty about text":

$I(X; Y) = H(Y) - H(Y|X)$

where $H$ is entropy.

The contrastive loss with $B$ negatives approximates:

$I(v_{\text{img}}; v_{\text{text}}) \geq \log(B) - \mathcal{L}_{\text{InfoNCE}}$

Why?

1. With $B$ candidates, random guessing has entropy $\log(B)$
2. Our loss reduces this to $\mathcal{L}$
3. The gap $\log(B) - \mathcal{L}$ is the information we've extracted

Practical implication: Larger batch sizes $B$ → better training signal (more hard negatives to contrast against).

Typical CLIP uses $B = 32,768$!

Common Mistakes

Why it feels right: Simple—just stack the vectors and process jointly.

Why it's wrong:

1. Dimension mismatch: Images might give2048-dim features, text 768-dim—raw concatenation loses structure
2. No alignment: Concatenation doesn't teach the model that "dog photo" and "dog word" represent the same concept
3. Not interpretable: Can't do retrieval (find matching images for text)

Fix: Use projection layers to map both to common dimension, then train with contrastive loss to align semantics.

Correct approach:

Image → CNN → Project to 512-dim → Normalize
Text → Transformer → Project to 512-dim → Normalize
Compute cosine similarity → Contrastive loss

Why it feels right: One less hyperparameter to tune.

Why it's wrong:

• $\tau=1$ makes the distribution too soft—model doesn't learn to be confident
• $\tau$ too small (e.g., 0.01) makes gradient vanishing (softmax saturates)

Example: With similarities [0.9, 0.3, 0.2]:

$\tau=1$: Softmax → [0.55, 0.24, 0.21] (only 55% on correct!) $\tau=0.1$: Softmax → [0.996, 0.002] (99.6% on correct)

Fix: Learn $\tau$ as a parameter (initialized ~0.07) or do grid search. CLIP learns it, starting from $\log(1/0.07) \approx 2.66$.

Why it feels right: Deep cross-attention between all image patches and all text tokens should capture fine-grained alignments.

Why it's partially wrong:

• Computational cost: $O(N_{\text{patches}} \times N_{\text{tokens}})$ attention is expensive
• Overfitting: On smaller datasets, complex fusion overfits
• Task dependent: Image-text retrieval only needs global similarity (simple embedings work). Visual QA needs fine-grained reasoning (complex fusion helps).

Fix: Match architecture complexity to task. For retrieval: dual encoders (CLIP). For generation: cross-attention (Flamingo, BLIP).

Implementation Details

Data Augmentation

Critical for contrastive learning—prevents shortcut learning (model matching images by low-level features like color, not semantics).

Augmentations:

• Random crop & resize: Forces model to recognize objects at different scales
• Color jitter: Prevents matching on color distribution
• Gaussian blur: Prevents matching on texture
• Horizontal flip: Spatial invariance

WHY augment both positive pairs identically? So the model learns invariant features—a cat is still a cat whether it's on the left or right of the image.

Batch Size Scaling

Contrastive loss needs large batches (many negatives).

Standard training: Batch size 256-512 CLIP training: Batch size 32,768 (!)

How? Distributed training with gradient accumulation:

For each micro-batch of size 256:
  1. Forward pass → local loss
  2. Accumulate gradients (don't update yet)
After 128 micro-batches (= 32,768 total):
  3. Synchronize gradients across GPUs
  4. Update parameters

WHY this works: The contrastive loss is additive over negatives—accumulating gradients is mathematically equivalent to one giant batch.

Imagine you're playing a matching game with picture cards and word cards. You need to learn which pictures match which words.

The computer does this by creating a "magic space" where it puts both pictures and words. If picture and word mean the same thing (like a photo of a cat and the word "cat"), the computer tries to put them close together in this magic space.

To learn this, it plays a game: "Here's a picture of a dog. Which word matches best: 'dog', 'cat', or 'car'?" If it picks 'dog', it gets a gold star! If it picks wrong, it gets feedback to move the dog picture closer to the 'dog' word.

After playing this game millions of times with different pictures and words, the computer becomes really good at understanding that pictures and words can mean the same thing—even for things it's never seen before! That's why these models can describe new photos or find pictures matching any description you type.

Active Recall

#flashcards/ai-ml

What is a multimodal vision-language model? :: A neural architecture with separate encoders for images and text that learn shared embedding space where cross-modal operations (retrieval, generation, reasoning) become possible through training on paired (image, text) data.

Derive the contrastive loss (InfoNCE) for one image :: Given image $i$ with matching text $t_i$ in a batch of $B$ pairs: $\mathcal{L}_i = -\log \frac{\exp(\text{sim}(v_i, v_{t_i}) / \tau)}{\sum_{j=1}^{B} \exp(\text{sim}(v_i, v_{t_j}) / \tau)}$ where $\tau$ is temperature. This is softmax cross-entropy treating the correct text as the label among $B$ candidates.

Connections

• 6.2.03-Transformer-architecture - Vision Transformers adapt self-attention for image patches
• 6.3.02-Self-supervised-learning - Contrastive learning is a form of self-supervision at massive scale
• 6.1.04-Word-embeddings - Joint embedings extend word embeddings to multiple modalities
• 6.4.01-Transfer-learning - Zero-shot classification is extreme transfer without fine-tuning
• 5.3.03-Cross-entropyloss - InfoNCE is softmax cross-entropy over batch negatives
• 6.2.05-Attention-mechanisms - Cross-attention enables fine-grained vision-language fusion
• 6.5.03-VisionTransformers - ViT serves as vision encoder in modern multimodal models
• 6.5.06-Text-to-image-generation - Inverse task: generate images from text using aligned embedings

Further Exploration

Research Papers:

• CLIP: Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021)
• ALIGN: Scaling Up Visual and Vision-Language Representation Learning (Jia et al., 2021)
• Flamingo: a Visual Language Model for Few-Shot Learning (Alayrac et al., 2022)
• BLIP-2: Bootstrapping Language-Image Pre-training (Li et al., 2023)

Key Insights:

1. Scale matters: CLIP's success came from 400M pairs, not novel architecture
2. Frozen pre-trained models: Modern approaches (Flamingo, BLIP-2) freeze unimodal experts and only train adapters—parameter efficient and leverages existing capabilities
3. Emergent abilities: Zero-shot transfer to tasks never explicitly trained on (OCR, counting, spatial reasoning)

Open Problems:

• Compositional reasoning: "a red cube on a blue sphere" vs "a blue cube on a red sphere"
• Counting: Current models struggle with "how many dogs?"
• Fine-grained alignment: Matching specific image regions to text phrases (grounding)
• Temporal reasoning: Extending to video-language (motion, events, causality)

Concept Map

Hinglish (regional understanding)