3.5.12 · AI-ML › Sequence Models

Overview

Real-world sequence modeling mein hume fixed-length inputs ki luxury bahut kam milti hai. Sentences ki length alag hoti hai, audio clips ki duration different hoti hai, aur time series ke spans irregular hote hain. Ye note neural networks mein variable-length sequences handle karne ki core techniques explore karta hai.

Core Approaches

1. Padding & Masking

Isko kyun karte hain: Modern deep learning frameworks data ko batches mein process karte hain efficiency ke liye. Ek batch tensor rectangular hona chahiye—sabhi sequences ka length dimension same hona zaroori hai.

Masked loss ki derivation: Maano hamare paas ek batch mein true lengths $\{L_1,L_2,..,L_B\}$ wali sequences hain, jo max length $L_{max}$ tak pad ki gayi hain.

Sequence $i$ ke liye ek mask define karte hain: $m_i(t)=\{\begin{array}{ll}1& \text{if }t\le L_i\\ 0& \text{if }t>L_i\end{array}$

Ye mask kyun? Hum chahte hain ki loss computation mein padding positions ko ignore kiya jaaye. Mask real tokens ke liye 1 hai aur padding ke liye 0.

Sequence $i$ ke liye masked loss banta hai: ${\mathcal{L}}_i=\frac{1}{{\sum }_{t=1}^{L_{max}}{m}_i(t)}{\sum }_{t=1}^{L_{max}}{m}_i(t)\cdot \ell (y_i(t),{\hat{y}}_i(t))$

Masks ke sum se divide kyun karte hain? Ye actual sequence length $L_i$ se normalize karta hai, padded length se nahi. Iske bina, shorter sequences ka loss artificially lower hota kyunki unke paas zyada zero-weighted terms hain.

Full batch ke liye: ${\mathcal{L}}_{batch}=\frac{1}{B}{\sum }_{i=1}^B{\mathcal{L}}_i$

Padded batch (max_len=4, pad_token=0):

[[1, 5, 7, 0],
 [2, 8, 3, 9],
 [4, 0, 0, 0]]

Mask matrix:

[[1, 1, 0],
 [1, 1, 1, 1],
 [1, 0, 0]]

Ye step kyun? Loss ya attention scores compute karte waqt, hum mask se multiply karte hain taaki padding positions zero ho jaayein. Jaise, agar model har position ke liye class probabilities predict karta hai, toh hum sirf wahan cross-entropy loss compute karte hain jahan mask=1 ho.

2. Packing & Unpacking (PyTorch)

Packing kyun? Padding batching ke liye zaroori hai, lekin RNNs ko padding tokens par computation waste nahi karni chahiye. PyTorch ka pack_padded_sequence efficient RNN computation ke liye padding hata deta hai, phir pad_packed_sequence baad ki layers ke liye padded format wapas restore karta hai.

Ye andar se kaise kaam karta hai:

Length ke hisaab se descending order mein sort ki gayi padded sequences:

Lengths: [4, 3, 1]
Paded:
t=0: [a₀, b₀, c₀]
t=1: [a₁, b₁, 0 ]
t=2: [a₂, b₂, 0 ]
t=3: [a₃, 0,  0 ]

Packed representation sabhi real tokens ko concatenate karta hai:

data: [a₀, b₀, c₀, a₁, b₁, a₂, b₂, a₃]
batch_sizes: [3, 2 1]

Ye ordering kyun? Time step $t$ par, batch_sizes[t] batata hai ki kitni sequences abhi "alive" hain (khatam nahi hui hain). RNN step $t$ par batch_sizes[t] elements process karta hai.

Padded input (batch_size=3, seq_len=5, embed_dim=10)

paded = torch.randn(3, 5, 10) lengths = torch.tensor([5, 3, 2])

Pack

packed = pack_padded_sequence(padded, lengths, batch_first=True)

packed.data.shape: (10, 10) ← 5+3+2=10 real elements

packed.batch_sizes: [3, 2, 1, 1, 1]

**`batch_sizes=[3,2,1,1,1]` kyun?**
- t=0: Teeno sequences active hain
- t=1: 2 sequences active hain (ek length 2 ke baad khatam ho gayi)
- t=2-4: 1 sequence active hai (ek length 3 ke baad khatam ho gayi)

Isse RNN dynamically har time step par batch size adjust kar leta hai—bahut bada speedup!

---

### 3. Dynamic Unrolling

> [!definition] Dynamic RNN Unrolling
> Fixed number of steps unroll karne ki jagah, ==dynamic control flow== (loops) use karo taaki har sequence uski actual length tak process ho sake.

**Mathematical formulation:**

Sequences ke ek batch ke liye, hum process karte hain:
$$h_i(t) = \text{RNN}(x_i(t), h_i(t-1)) \quad \text{for } t = 1, 2, .., L_i$$

jahan $L_i$ sequence $i$ ki true length hai, koi fixed $L_{max}$ nahi.

**Ye kyun kaam karta hai:** Modern frameworks (TensorFlow ka dynamic_rnn, PyTorch ka native RNN with packed sequences) in dynamic loops ko efficiently compile karte hain.

> [!formula] Hidden State Update (Concrete Example)
> Ek simple RNN ke liye:
> $$h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$$

**First principles se derivation:**
- **Hum kya compute kar rahe hain?** Ek nayi hidden state jo current input aur previous state par depend karti hai
- **tanh kyun?** Values ko [-1, 1] mein squash karta hai, exploding activations rokta hai
- **Do weight matrices kyun?** $W_{xh}$ input dimension ko hidden dimension mein transform karta hai; $W_{hh}$ previous hidden state ko transform karta hai (recurrence)

Variable lengths ke liye, hum simply sequence $i$ ke liye $t > L_i$ par update karna band kar dete hain. Key insight: **weights $W_{h}, W_{xh}$ sabhi time steps aur sabhi sequence lengths par shared hain**—yahi parameter efficiency hai.

---

### 4. Attention Mechanisms (Length-Agnostic)

> [!intuition] Attention Variable Lengths Ko Naturally Kyun Handle Karta Hai
> Attention ko sequence length se koi farq nahi padta kyunki ye ==sabhi positions par ek weighted sum== hai. Chahe 5 tokens hon ya 500, attention mechanism compute karta hai:
> $$\text{output}_i = \sum_{j=1}^{L} \alpha_{ij} v_j$$

jahan $L$ actual sequence length hai (math mein padding ki zaroorat nahi).

**Scaled dot-product attention ki derivation:**

Diya gaya:
- Query: $q \in \mathbb{R}^{d_k}$ (jo hum dhundh rahe hain)
- Keys: $K \in \mathbb{R}^{L \times d_k}$ (jo available hai)
- Values: $V \in \mathbb{R}^{L \times d_v}$ (jo hum return karte hain)

Step 1: Similarity scores compute karo
$$s_j = q^T k_j \quad \text{for } j=1,...,L$$

**Dot product kyun?** Query aur key ke beech alignment measure karta hai. High dot product = embedding space mein similar directions.

Step 2: Scores ko scale karo
$$s_j' = \frac{q^T k_j}{\sqrt{d_k}}$$

**$\sqrt{d_k}$ se scale kyun karte hain?** High-dimensional vectors ke liye, dot products magnitude mein bade ho jaate hain. Agar $q$ aur $k$ ki unit variance hai, toh $q^T k$ ki variance $d_k$ hogi. $\sqrt{d_k}$ se divide karne par variance 1 ho jaati hai, softmax saturation ruk jaata hai.

**Mathematical proof:**
Agar $q, k \sim \mathcal{N}(0, I)$, toh:
$$\text{Var}(q^T k) = \text{Var}\left(\sum_{i=1}^{d_k} q_i k_i\right) = \sum_{i=1}^{d_k} \text{Var}(q_i)\text{Var}(k_i) = d_k$$

Toh $\text{Var}\left(\frac{q^T k}{\sqrt{d_k}}\right) = \frac{d_k}{d_k} = 1$.

Step 3: Attention weights compute karo
$$\alpha_j = \frac{\exp(s_j')}{\sum_{k=1}^{L} \exp(s_k')}$$

**Softmax kyun?** Scores ko probability distribution mein convert karta hai (sum 1, sab positive). $\exp$ differences ko emphasize karta hai.

Step 4: Weighted sum
$$\text{output} = \sum_{j=1}^{L} \alpha_j v_j$$

**Variable lengths ke liye key insight:** Ye poori computation kisi bhi $L$ ke liye kaam karti hai. Attention math mein padding ki zaroorat nahi! Hume sirf masks chahiye taaki padding positions par attention na jaaye.

> [!example] Attention with Masking
> Maano hamare paas 2 sequences hain, lengths [3, 2], length 3 tak padded:
> ```
> Sequence 1: [a, b, c]    ← real
> Sequence 2: [d, e, PAD]  ← PAD ko ignore karna chahiye
> ```

Masking se pehle attention scores (seq 2 ki position 1 par query):
$$s = [s_d, s_e, s_{PAD}]$$

Softmax se pehle $s_{PAD} = -\infty$ set karke mask apply karo:
$$s_{masked} = [s_d, s_e, -\infty]$$

**$-\infty$ kyun?** Softmax ke baad: $\exp(-\infty) = 0$, toh padding ko zero attention weight milta hai.

Softmax ke baad:
$$\alpha =[\alpha_d, \alpha_e, 0]$$

Output naturally padding position ko ignore kar leta hai.

---

## Common Strategies Compared

| Method | Memory Computation | Complexity |
|--------|---------|------------|
| Padding + Masking | High (padding store hoti hai) | Padding par cycles waste hote hain | Implement karna simple hai |
| Packed Sequences | Low (padding store nahi hoti) | Efficient (padding skip hoti hai) | Moderate (packing/unpacking) |
| Dynamic Unrolling | Medium | Efficient | Framework-dependent |
| Attention | Medium | Length-independent per head | High (quadratic in length) |

---

## Implementation Patterns

> [!example] PyTorch Packed Sequence
> ```python
# Assume: batch of sequences, sorted descending by length
# embedded: (batch, max_len, embed_dim)
# lengths: [7, 5, 3]

packed = pack_padded_sequence(embedded, lengths, batch_first=True)
# packed.data: (15, embed_dim) ← 7+5+3=15

packed_output, hidden = rnn(packed)
# packed_output.data: (15, hidden_dim)

output, _ = pad_packed_sequence(packed_output, batch_first=True)
# output: (batch, max_len, hidden_dim) ← wapas padded format mein

Ye sequence kyun?

1. Pack padding hata deta hai → RNN sirf real tokens process karta hai
2. RNN packed format par operate karta hai (internally variable batch sizes handle karta hai)
3. Unpack baad ki layers ke liye rectangular shape restore karta hai (attention, linear layers, etc.)

Ye sahi kyun lagta hai: "Maine sequences pad kar diye, toh sab kuch kaam karna chahiye."

Problem: Model padding tokens predict karna seekh leta hai! Isse:

• Real tokens ka loss signal dilute ho jaata hai
• Model PAD accurately predict karke "cheat" karne lagta hai
• Perplexity aur doosre metrics kharab ho jaate hain

Sahi approach:

mask = (targets != pad_token).float()  # real ke liye 1, padding ke liye 0
loss = (criterion(predictions, targets) * mask).sum() / mask.sum()

Fix kyun kaam karta hai: Hum padding positions par loss zero kar dete hain, phir real tokens ki sankhya se normalize karte hain.

Ye sahi kyun lagta hai: "Function koi bhi order handle kar sakta hai."

Problem: pack_padded_sequence assume karta hai ki lengths descending sorted hain. Warna packed representation galat hoga—batch_sizes bhi galat honge.

Fix:

lengths, sort_idx = lengths.sort(descending=True)
embedded = embedded[sort_idx]
packed = pack_padded_sequence(embedded, lengths, batch_first=True)
# ... process ...
# Agar zaroorat ho toh outputs unsort karo: output = output[sort_idx.argsort()]

Option 1 (Padding): Sabko boldo ki exactly 5 pages submit karo. Agar tumne sirf 2 pages likhe hain, toh 3 blank pages add karo. Lekin phir tumhe un blank pages ko bhi palat na padega—time waste! Aur average essay quality calculate karte waqt, tumhe blank pages skip karna yaad rakhna hoga.

Option 2 (Packing): Sabki real pages ek stack mein collect karo, aur ek note rakh lo: "Student A ne 1 page likhi, Student B ne 5 pages likhi, Student C ne 2 pages likhi." Ab tum sirf real content padhte ho! Jab ho jaao, zaroorat pade toh unhe wapas alag essays mein organize kar sakte ho.

Option 3 (Attention): Page-by-page order mein padhne ki jagah, tum saari pages padhte ho aur decide karte ho ki har question ke liye kaun se parts sabse important hain. Koi farq nahi padta ki page 1 hai ya page 5—tum sirf relevant cheez par focus karte ho. Kuch pages blank ho sakti hain (padding), toh unhe bilkul ignore karo.

Neural networks ko variable-length sentences ke saath yahi problem hoti hai. Hum ye tricks isliye use karte hain taaki network blank pages process karne mein time waste na kare!

Yaad rakho: "PAM tumhari sequences efficiently pack karta hai!"

Connections

• Recurrent Neural Networks - core architecture jisko variable-length handling chahiye
• Attention Mechanism - alternative paradigm jo lengths naturally handle karta hai
• Transformers - positional encoding + attention use karte hain, koi recurrence nahi chahiye
• Batch Processing - kyun variable lengths batching ko complicated banate hain
• Sequence-to-Sequence Models - encoder-decoder with different input/output lengths
• LSTM and GRU - specific RNN variants jo ye techniques use karte hain
• Masking in Deep Learning - sequences se aage general concept

#flashcards/ai-ml

Sequence models mein padding kyun chahiye hoti hai? :: Neural networks batches ko rectangular tensors ke roop mein process karte hain—efficient matrix operations ke liye batch ke sabhi sequences ka length dimension same hona zaroori hai.

Concept Map