Bahdanau and Luong attention — Hinglish notes

3.5.10 · AI-ML › Sequence Models

Intuition Core Problem: Fixed-Length Bottleneck

Imagine karo ki tum poori ek book ko ek sentence mein summarize kar rahe ho, aur phir sirf usi sentence ke basis par review likh rahe ho. Tum kaafi zaroori details kho doge! Early seq2seq models bilkul yahi karte the — poore input sequence ko ek fixed-size vector mein compress kar dete the (encoder ki final hidden state). Lambe sequences ke liye yeh bottleneck catastrophic information loss ka karan banta tha.

Attention iska solution hai: Sabkuch ek bottleneck se force karne ki jagah, decoder ko saare encoder states par "nazar daalane" do aur dynamically har output word ke liye sabse relevant parts par focus karne do. Yeh bilkul aisa hai jaise review likhte waqt poori book khuli ho, aur tum zaroorat ke hisaab se relevant pages par ja sako.

Bahdanau aur Luong Attention kya hain?

Bahdanau attention (jise additive attention bhi kehte hain) aur Luong attention (jise multiplicative attention bhi kehte hain) — yeh do pioneering mechanisms hain jo neural sequence models ko allow karte hain ki woh har output token generate karte waqt input ke alag-alag parts par selectively focus karein.

Fundamental difference: YEH ki woh decoder aur encoder states ke beech alignment scores KAISE compute karte hain.

Shared Framework

Dono mechanisms is pattern ko follow karte hain:

Alignment Scoring: Compute karo ki har encoder hidden state current decoder state ke liye kitni relevant hai
Attention Weights: Scores ko ek probability distribution mein convert karo (softmax)
Context Vector: Attention weights use karke encoder states ka weighted sum
Output Generation: Agla token predict karne ke liye context vector (+ decoder state) use karo

Bahdanau Attention (Additive/Concat)

First Principles se Derivation

Setup:

Encoder hidden states: $h_{1}, h_{2}, \dots, h_{T}$ (har ek $\in R^{d_{h}}$ )
Step $t$ par decoder state: $s_{t - 1} \in R^{d_{s}}$
Humein output $y_{t}$ generate karna hai

Step 1: Alignment Score Function

Concatenate kyun? Hum decoder state aur har encoder state ke beech interaction capture karna chahte hain. Simple dot product (Luong) assume karta hai ki spaces already aligned hain; Bahdanau ek alignment function seekhta hai.

$e_{t i} = v_{a}^{T} tanh (W_{a} s_{t - 1} + U_{a} h_{i})$

JAHAN:

$W_{a} \in R^{d_{a} \times d_{s}}$ decoder state ko project karta hai
$U_{a} \in R^{d_{a} \times d_{h}}$ encoder state ko project karta hai
$v_{a} \in R^{d_{a}}$ alignment weight vector hai
$d_{a}$ attention dimension hai (hyperparameter, aksar 128-512)

Yeh form kyun?

Dono states ko ek common space mein project karo ( $d_{a}$ )
$tanh$ non-linearity provide karta hai (complex alignments seekhne deta hai)
$v_{a}^{T}$ scalar score tak reduce karta hai

Yeh $d_{a}$ hidden units wala ek single-layer feedforward network hai.

Step 2: Attention Weights

Scores ko probability distribution mein convert karo (ZAROORI hai ki sum 1 ho, sab positive):

$α_{t i} = \frac{e x p ( e _{t i} )}{\sum _{j = 1}^{T} e x p ( e _{t j} )}$

Softmax kyun? Ensure karta hai ki $\sum_{i} α_{t i} = 1$ (proper probability distribution) aur high-scoring positions ko emphasize karta hai (exponential amplification).

Step 3: Context Vector

Saare encoder states ka weighted average:

$c_{t} = \sum_{i = 1}^{T} α_{t i} h_{i}$

Yeh input ka "summary" hai jo $y_{t}$ predict karne ke liye relevant hai.

Step 4: Decoder Update

Bahdanau RNN step se PEHLE context concatenate karta hai:

$\tilde{y}_{t - 1} = [y_{t - 1}; c_{t}]$

$s_{t} = GRU (\tilde{y}_{t - 1}, s_{t - 1})$

$p (y_{t} ∣ y_{< t}, x) = softmax (W_{o} s_{t})$

Pehle concatenate kyun karte hain? Yeh input-feeding hai: context RNN state update ko directly influence karta hai, jisse attention information ka hidden state evolution par zyada direct control hota hai.

Luong Attention (Multiplicative)

Teen Scoring Variants

Luong ne teen alignment functions propose kiye:

1. Dot Product (sabse simple): $score (s_{t}, h_{i}) = s_{t}^{T} h_{i}$

Yeh kyun kaam karta hai: Agar decoder aur encoder states same semantic space mein hain, toh high similarity = high relevance. Koi parameters nahi!

Limitation: $d_{s} = d_{h}$ zaroori hai.

2. General/Bilinear (sabse common): $score (s_{t}, h_{i}) = s_{t}^{T} W_{a} h_{i}$

JAHAN $W_{a} \in R^{d_{s} \times d_{h}}$ ek learned weight matrix hai.

Yeh better kyun hai? Encoder space ko decoder space mein map karna seekhta hai. $d_{s} \neq = d_{h}$ ko handle karta hai. Phir bhi computationally sasta hai (single matrix multiply).

3. Concat (Bahdanau se milta-julta): $score (s_{t}, h_{i}) = v_{a}^{T} tanh (W_{a} [s_{t}; h_{i}])$

Complete Luong Attention (General)

Step 1: Pehle Decoder RNN (KEY DIFFERENCE)

$s_{t} = RNN (y_{t - 1}, s_{t - 1})$

Pehle attention ke bina decoder state compute karo.

Step 2: Alignment Scores

$e_{t i} = s_{t}^{T} W_{a} h_{i}$

Step 3: Attention Weights

$α_{t i} = \frac{e x p ( e _{t i} )}{\sum _{j = 1}^{T} e x p ( e _{t j} )}$

Step 4: Context Vector

$c_{t} = \sum_{i = 1}^{T} α_{t i} h_{i}$

Step 5: Attentional Hidden State

Key innovation — decoder state aur context ko combine karo:

$\tilde{s}_{t} = tanh (W_{c} [s_{t}; c_{t}])$

JAHAN $W_{c} \in R^{d_{s} \times (d_{s} + d_{h})}$ .

Kyun? Yeh learned combination decide karta hai ki context par kitna rely karna hai versus decoder state par. $tanh$ output ko bound karta hai, explosion rokta hai.

Step 6: Output

$p (y_{t} ∣ y_{< t}, x) = softmax (W_{o} \tilde{s}_{t})$

Side-by-Side Comparison

Aspect	Bahdanau (Additive)	Luong (Multiplicative)
Kab compute hoti hai	Decoder RNN step se pehle	Decoder RNN step ke baad
Decoder state use karta hai	$s_{t - 1}$ (previous)	$s_{t}$ (current)
Score function	$v_{a}^{T} tanh (W_{a} s + U_{a} h)$	$s^{T} W_{a} h$ (general)
Complexity	$O (T \cdot d_{a} \cdot (d_{s} + d_{h}))$	$O (T \cdot d_{s} \cdot d_{h})$
Parameters	Zyada (3 weight matrices)	Kam (1-2 weight matrices)
Input feeding	Haan (context RNN ko fed hoti hai)	Nahi (context baad mein combine hoti hai)
Speed	Slow	Fast
Performance	Lambe sequences par thoda better	Competitive, aksar preferred

Timing ka difference kyun matter karta hai:

Bahdanau: Context hidden state evolution ko influence karta hai → zyada integrated reasoning
Luong: Cleaner separation, interpret aur implement karna asaan

Worked example Example 1: Translation Step

Task: "The cat sat" ka anuvad → "Le chat s'est assis" Encoder states: $h_{1}$ (The), $h_{2}$ (cat), $h_{3}$ (sat) Current output: word2 generate kar rahe hain (French: "chat")

Bahdanau Process:

Previous decoder state: $s_{1}$ ("Le" generate karne ke baad)
Scores compute karo:

$e_{21} = v_{a}^{T} tanh (W_{a} s_{1} + U_{a} h_{1}) = 0.5$
$e_{22} = v_{a}^{T} tanh (W_{a} s_{1} + U_{a} h_{2}) = 2.1$ ← HIGH
$e_{23} = v_{a}^{T} tanh (W_{a} s_{1} + U_{a} h_{3}) = 0.3$

Softmax: $α_{21} = 0.12, α_{22} = 0.79, α_{23} = 0.09$
Context: $c_{2} = 0.12 h_{1} + 0.79 h_{2} + 0.09 h_{3}$ (mostly "cat")
Update: $s_{2} = GRU ([emb ("Le"); c_{2}], s_{1})$
Output: "chat" high probability ke saath

Yeh steps kyun? Model backprop ke through "chat" generate karte waqt "cat" ko attend karna seekhta hai. Alignment function translation mein monotonic patterns discover karta hai.

Worked example Example 2: Luong Attention Calculation

Dimension setup: $d_{s} = 256$ , $d_{h} = 512$ , $T = 5$ input tokens Step-by-step:

Decoder $s_{3} \in R^{256}$ compute karta hai (RNN step ke baad)
Score calculation (general):

$W_{a} \in R^{256 \times 512}$
Har $i$ ke liye: $e_{3 i} = s_{3}^{T} W_{a} h_{i}$ (256×1)ᵀ × (256×512) × (512×1) = scalar
Results: $[e_{31}, \dots, e_{35}] = [1.2, 3.5, 0.8, 2.1, 0.5]$

Softmax: $α_{32} = \frac{e ^{3.5}}{e ^{1.2} + e ^{3.5} + e ^{0.8} + e ^{2.1} + e ^{0.5}} = 0.64$ (mostly position 2 ko attend karta hai)
Context: $c_{3} = \sum_{i = 1}^{5} α_{3 i} h_{i} \in R^{512}$
Combine: $\tilde{s}_{3} = tanh (W_{c} [s_{3}; c_{3}])$ jahan $[s_{3}; c_{3}] \in R^{768}$
Final: $W_{c} \in R^{256 \times 768}$ se $\tilde{s}_{3} \in R^{256}$ milta hai Concatenate aur project kyun karte hain? Concatenation dono decoder reasoning aur input context preserve karta hai. Projection ( $W_{c}$ ) optimal blend seekhta hai — shayad is example ke liye 70% decoder state, 30% context.

Worked example Example 3: Computational Efficiency

Setup:

Input sequence length: $T = 50$
$d_{s} = d_{h} = 512$ , $d_{a} = 256$ (Bahdanau)
Batch size: 32 Har decoder step par Bahdanau ka cost:
Score computation: alignment network ke through $T$ forward passes
Har ek: $W_{a} s_{t - 1}$ : (256×512) × (512×1) = 131K ops
Plus: $U_{a} h_{i}$ : (256×512) × (512×1) = 131K ops
Plus: $v_{a}^{T} tanh (...)$ : 256 ops
Total per position: ~262K ops
$T = 50$ ke liye: 13.1M ops
Softmax: $50$ exponentials + normalization ≈ 150 ops
Context: 50 vectors ka weighted sum ≈ 25K ops
Total: ~13.1M ops per decoder step

Har decoder step par Luong ka cost (general):

Score computation: $T$ bilinear products
$W_{a} h_{i}$ : ek baar precompute karo: 50× (512×512) = 13.1M ops
$s_{t}^{T} (W_{a} h_{i})$ : 50 × 512 = 25K ops
Total: 13.1M (ek baar) + 25K (har step)
Softmax: 150 ops
Context: 25K ops
Combine: $W_{c} [s_{t}; c_{t}]$ : (512×1024)×(1024×1) = 524K ops
Total: ~575K ops per step (precomputation ke baad)

Luong 20× faster kyun hai: Matrix precomputation! $W_{a} h_{i}$ $t$ se independent hai, isliye saare decoder steps ke liye EK BAAR compute karo.

Common mistake Mistake 1: Padding Mask Bhool Jana

Galat approach: Softmax saare $T$ positions par apply karo, including padding tokens.

# WRONG
scores = decoder_state @ encoder_states.T  # (T,)
attention_weights = softmax(scores)  # padding shamil hai!

Yeh sahi lagta hai kyun: Softmax normalize karta hai, toh padding par small scores ko small weights milenge.

Yeh galat kyun hai: Padding par tiny attention bhi context vector ko meaningless information se corrupt karta hai. 40 real tokens aur 10 padding wali sequence ke liye, woh 10 positions shayad 0.01 weight har ek ko milega = 10% total mass waste.

Fix: Softmax se pehle padding positions ko $- \infty$ se mask karo:

# CORRECT
scores = decoder_state @ encoder_states.T
scores = scores.masked_fill(padding_mask, float('-inf'))
attention_weights = softmax(scores)  # padding ko exactly 0 weight milta hai

Yeh kyun kaam karta hai: $exp (- \infty) = 0$ , isliye padding positions context mein kuch contribute nahi karte.

Common mistake Mistake 2: Luong Architecture ke liye Bahdanau Score Use Karna

Galat approach: Additive scoring ( $v^{T} tanh (...)$ ) use karo lekin attention decoder RNN step ke BAAD apply karo.

Yeh sahi lagta hai kyun: "Bahdanau scoring zyada expressive hai, toh yeh hamesha better hona chahiye."

Problem: Bahdanau ka score function $s_{t - 1}$ expect karta hai (current input dekhne se PEHLE ki state). Jab tum $s_{t}$ use karte ho (input dekhne ke baad), toh alignment kam meaningful ho jaata hai kyunki $s_{t}$ already "jaanta hai" ki kaun sa output generate karna hai.

Yeh kyun matter karta hai:

Bahdanau: "Ab tak jo generate kiya hai ( $s_{t - 1}$ ) uske hisaab se, mujhe kaun sa input dekhna chahiye?"
Luong: "Mera RNN kya sochta hai ( $s_{t}$ ) uske hisaab se, mujhe input context se kaise refine karna chahiye?"

Attention computation ki timing score function design se coupled hai.

Fix: Scoring aur timing ko match karo:

Additive score → RNN se pehle attention (Bahdanau)
Multiplicative score → RNN ke baad attention (Luong)

Common mistake Mistake 3: Dot-Product Attention mein Dimension Mismatch

Galat setup:

encoder_dim = 512  # bidirectional LSTM: 256 forward + 256 backward
decoder_dim = 256  # unidirectional LSTM
 
score = decoder_state @ encoder_state.T  # ERROR: (256,) @ (512,)

Yeh sahi lagta hai kyun: "Dot product parameter-free aur fast hai!"

Yeh fail kyun karta hai: Matrix dimension mismatch. $(1 \times 256)$ ko $(1 \times 512)^{T} = (512 \times 1)$ se multiply nahi kar sakte.

Steel-man: Shayad tum socho "main sirf encoder ke pehle 256 dims use karunga," lekin isse aadhi information discard ho jaati hai.

Fix 1: General/bilinear Luong use karo:

W_a = nn.Linear(encoder_dim, decoder_dim)  # 512 → 256
score = decoder_state @ W_a(encoder_state).T  # Kaam karta hai!

Fix 2: Decoder ko encoder space mein project karo:

W_d = nn.Linear(decoder_dim, encoder_dim)  # 256 → 512
score = W_d(decoder_state) @ encoder_state.T

Fix 3: Shuruaat se hi dimensions match karo (decoder ko 512-dim banao).

Local vs Global Attention (Luong Extension)

Luong ne complexity kam karne ke liye local attention bhi introduce ki:

Global attention: SAARE encoder positions ko attend karo (ab tak hum yahi describe kar rahe the).

Complexity: $O (T \cdot d)$ per decoder step
Problem: Lambe sequences ke liye expensive

Local attention: Predicted position ke aaspaas ek WINDOW ko attend karo.

Alignment position predict karo: $p_{t} = S \cdot sigmoid (v_{p}^{T} tanh (W_{p} s_{t}))$ jahan $S$ source length hai
Window define karo: $[p_{t} - D, p_{t} + D]$ (e.g., $D = 10$ )
Sirf window ke andar attention compute karo
Gaussian weighting apply karo: $α_{t i} = softmax (e_{t i}) \cdot exp (- \frac{( i - p _{t} ) ^{2}}{2 σ ^{2}})$

Gaussian kyun? Predicted center se door attention smoothly decay karo, sharp cutoffs rokne ke liye.

Result: Complexity $O (T)$ se $O (D)$ per step tak girti hai, jahan $D ≪ T$ .

Implementation Considerations

Batching:

# Encoder states: (batch, seq_len, hidden_dim)
# Decoder state: (batch, hidden_dim)
 
# Broadcasting for efficient score computation:
decoder_expanded = decoder_state.unsqueeze(1)  # (batch, 1, hidden_dim)
scores = (decoder_expanded @ encoder_states.transpose(1,2)).squeeze(1)
# Result: (batch, seq_len)

Numerical stability: Bade $T$ ke liye raw softmax ki jagah log_softmax + exp use karo:

log_alpha = torch.log_softmax(scores, dim=1)
alpha = torch.exp(log_alpha)

Teacher forcing: Training ke dauran, model kuch aur predict kare tab bhi ground-truth $y_{t - 1}$ use karo. Yeh early training stabilize karta hai lekin test time par exposure bias cause kar sakta hai.

Recall 12 saal ke bachche ko samjhao

Imagine karo tum English se Spanish mein ek sentence translate kar rahe ho, word by word. Purana tarika yeh tha: poori English sentence ek baar padho, kitaab band karo, aur phir sab yaad rakhne ki koshish karte hue Spanish likhna shuru karo. Agar sentence lamba ho, toh zaroori parts bhool jaoge!

Attention matlab hai ki English kitaab khuli rakhne ki permission milna. Har Spanish word likhte waqt, tum English sentence par nazar daal sakte ho aur us part par focus kar sakte ho jo abhi sabse helpful hai.

Bahdanau matlab hai kitaab dekhna PEHLE, phir agla Spanish word ke baare mein sochna. Pehle English check karo, phir socho.

Luong matlab hai pehle sochna ki agla Spanish word kya ho sakta hai, PHIR English kitaab check karo ki sahi track par ho.

Dono bahut acche kaam karte hain! Luong thoda faster hai kyunki woh English kitaab ke kuch hisson ko pehle se padh sakta hai, jabki Bahdanau har baar fresh padhta hai.

"Attention weights" highlighting jaisi hain: tum English words ko highlight karte ho jo current Spanish word ke liye sabse zyada matter karte hain. Kabhi ek word bright highlight hota hai (90% attention), kabhi kuch words par spread hota hai.

Connections

3.5.1-Sequence-to-Sequence-Models: Dono attention mechanisms seq2seq architecture extend karte hain
3.5.9-AttentionMechanism-Overview: Yeh general attention ke specific implementations hain
3.5.11-Self-Attention-and-Transformers: Transformers inhi ideas se evolve hue, self-attention use karke
3.5.8-Encoder-Decoder-Architecture: Attention encoder aur decoder ko bridge karta hai
2.4.7-Softmax-Function: Softmax scores ko probability distribution mein convert karta hai
3.3.5-Vanishing-and-Exploding-Gradients: Attention shorter gradient paths provide karke help karta hai
4.2.3-Beam-Search: Attention weights decoding mein beam search guide karte hain
5.1.2-Word-Embeddings: Attention embedding space mein operate karta hai

#flashcards/ai-ml

Bahdanau aur Luong attention timing ka key difference kya hai? :: Bahdanau attention decoder RNN step se PEHLE compute karta hai (uses $s_{t - 1}$ ), jabki Luong attention decoder RNN step ke BAAD compute karta hai (uses $s_{t}$ ).

Bahdanau attention score function kya hai?

e_{t i} = v_{a}^{T} tanh (W_{a} s_{t - 1} + U_{a} h_{i})

— ek additive/concat mechanism jo tanh nonlinearity wale feedforward network use karta hai.

Luong general attention score function kya hai?

e_{t i} = s_{t}^{T} W_{a} h_{i}

— ek bilinear/multiplicative mechanism jo encoder space ko decoder space mein map karna seekhta hai.

Bahdanau attention mein input feeding kya hai?

Context vector

c_{t}

ko input embedding ke saath concatenate karke decoder RNN ko feed kiya jaata hai:

s_{t} = RNN ([y_{t - 1}; c_{t}], s_{t - 1})

. Isse attention hidden state evolution ko directly influence kar sakti hai.

Luong attention context ko decoder state ke saath kaise combine karta hai?

Yeh ek attentional hidden state banata hai:

\tilde{s}_{t} = tanh (W_{c} [s_{t}; c_{t}])

jo decoder state aur context vector ko ek learned projection ke through combine karta hai.

Teen Luong attention scoring variants kya hain?

1) Dot:

s_{t}^{T} h_{i}

(koi parameters nahi, same dims zaroori), 2) General:

s_{t}^{T} W_{a} h_{i}

(sabse common), 3) Concat:

v_{a}^{T} tanh (W_{a} [s_{t}; h_{i}])

(Bahdanau se milta-julta).

Luong attention typically Bahdanau se faster kyun hota hai?

Luong

W_{a} h_{i}

ko saare encoder positions ke liye ek baar precompute kar sakta hai, kyunki yeh decoder state se independent hai. Bahdanau ko har decoder step par har encoder position ke liye additive function alag se compute karna padta hai.

Alignment scores se attention weights ka formula kya hai?

α_{t i} = \frac{e x p ( e _{t i} )}{\sum _{j = 1}^{T} e x p ( e _{t j} )}

— softmax normalization se ek probability distribution banta hai jo sum 1 hoti hai.

Dono mechanisms mein context vector ka formula kya hai?

c_{t} = \sum_{i = 1}^{T} α_{t i} h_{i}

— attention weights use karke saare encoder hidden states ka weighted sum.

Attention compute karne se pehle padding positions ko kyun mask karna zaroori hai?

Unmasked padding ko small but non-zero attention weights milte hain, probability mass waste hoti hai aur context vector meaningless information se corrupt hoti hai. Softmax se pehle

- \infty

se mask karne se padding positions ko exactly 0 weight milta hai.

Local attention (Luong extension) kya hai?

Ek optimization jo predicted alignment position

p_{t}

ke aaspaas sirf ek window

[p_{t} - D, p_{t} + D]

ko attend karta hai, complexity

O (T)

O (D)

per decoder step tak kam kar deta hai.

Attention mechanisms ka key architectural motivation kya hai?

Seq2seq models mein fixed-length bottleneck problem solve karna — poore input ko ek vector mein compress karne ki jagah, attention decoder ko dynamically saare encoder states access karne deta hai.

Bahdanau attention mein kitni weight matrices hoti hain?

Teen:

W_{a} \in R^{d_{a} \times d_{s}}

decoder projection ke liye,

U_{a} \in R^{d_{a} \times d_{h}}

encoder projection ke liye, aur

v_{a} \in R^{d_{a}}

alignment vector ke liye. Total parameters:

d_{a} (d_{s} + d_{h} + 1)

Luong general attention mein kitni weight matrices hoti hain?

Do:

W_{a} \in R^{d_{s} \times d_{h}}

scoring ke liye (parameters:

d_{s} \cdot d_{h}

) aur

W_{c} \in R^{d_{s} \times (d_{s} + d_{h})}

context aur decoder state combine karne ke liye (parameters:

d_{s} (d_{s} + d_{h})

Bahdanau score function mein tanh nonlinearity kyun use hoti hai?

Alignment function ko decoder aur encoder states ke beech complex, non-linear relationships seekhne dene ke liye. Nonlinearity ke bina yeh simple linear transformation tak collapse ho jaata.