Bahdanau and Luong attention

What Are Bahdanau and Luong Attention?

Bahdanau attention (also called additive attention) and Luong attention (also called multiplicative attention) are two pionering mechanisms that allow neural sequence models to selectively focus on different parts of the input when generating each output token.

The fundamental difference: HOW they compute alignment scores between decoder and encoder states.

The Shared Framework

Both mechanisms follow this pattern:

Alignment Scoring: Compute how relevant each encoder hidden state is to the current decoder state
Attention Weights: Convert scores to a probability distribution (softmax)
Context Vector: Weighted sum of encoder states using attention weights
Output Generation: Use context vector (+ decoder state) to predict next token

Bahdanau Attention (Additive/Concat)

Derivation from First Principles

Setup:

Encoder hidden states: $\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_T$ (each $\in \mathbb{R}^{d_h}$ )
Decoder state at step $t$ : $\mathbf{s}_{t-1} \in \mathbb{R}^{d_s}$
We need to generate output $y_t$

Step 1: Alignment Score Function

WHY concatenate? We want to capture the interaction between decoder state and each encoder state. A simple dot product (Luong) assumes the spaces are already aligned; Bahdanau learns an alignment function.

$e_{ti} = v_a^T \tanh(W_a \mathbf{s}_{t-1} + U_a \mathbf{h}_i)$

WHERE:

$W_a \in \mathbb{R}^{d_a \times d_s}$ projects decoder state
$U_a \in \mathbb{R}^{d_a \times d_h}$ projects encoder state
$v_a \in \mathbb{R}^{d_a}$ is the alignment weight vector
$d_a$ is the attention dimension (hyperparameter, often 128-512)

Why this form?

Project both states into a common space ( $d_a$ )
$\tanh$ provides non-linearity (allows learning complex alignments)
$v_a^T$ reduces to scalar score

This is a single-layer feedforward network with $d_a$ hidden units.

Step 2: Attention Weights

Convert scores to probability distribution (MUST sum to 1, all positive):

$\alpha_{ti} = \frac{\exp(e_{ti})}{\sum_{j=1}^{T} \exp(e_{tj})}$

WHY softmax? Ensures $\sum_i \alpha_{ti} = 1$ (proper probability distribution) and emphasizes high-scoring positions (exponential amplification).

Step 3: Context Vector

Weighted average of all encoder states:

$\mathbf{c}_t = \sum_{i=1}^{T} \alpha_{ti} \mathbf{h}_i$

This is the "summary" of input that's relevant for predicting $y_t$ .

Step 4: Decoder Update

Bahdanau concatenates context BEFORE the RNN step:

$\tilde{\mathbf{y}}_{t-1} = [\mathbf{y}_{t-1}; \mathbf{c}_t]$

$\mathbf{s}_t = \text{GRU}(\tilde{\mathbf{y}}_{t-1}, \mathbf{s}_{t-1})$

$p(y_t | y_{<t}, \mathbf{x}) = \text{softmax}(W_o \mathbf{s}_t)$

Why concatenate before? This is input-feeding: the context influences the RNN state update itself, giving attention information more direct control over the hidden state evolution.

Luong Attention (Multiplicative)

Three Scoring Variants

Luong proposed three alignment functions:

1. Dot Product (simplest): $\text{score}(\mathbf{s}_t, \mathbf{h}_i) = \mathbf{s}_t^T \mathbf{h}_i$

WHY this works: If decoder and encoder states are in the same semantic space, high similarity = high relevance. No parameters!

Limitation: Requires $d_s = d_h$ .

2. General/Bilinear (most common): $\text{score}(\mathbf{s}_t, \mathbf{h}_i) = \mathbf{s}_t^T W_a \mathbf{h}_i$

WHERE $W_a \in \mathbb{R}^{d_s \times d_h}$ is a learned weight matrix.

WHY better? Learns to map encoder space to decoder space. Handles $d_s \neq d_h$ . Still computationally cheap (single matrix multiply).

3. Concat (similar to Bahdanau): $\text{score}(\mathbf{s}_t, \mathbf{h}_i) = v_a^T \tanh(W_a[\mathbf{s}_t; \mathbf{h}_i])$

Complete Luong Attention (General)

Step 1: Decoder RNN First (KEY DIFFERENCE)

$\mathbf{s}_t = \text{RNN}(\mathbf{y}_{t-1}, \mathbf{s}_{t-1})$

Compute decoder state WITHOUT attention first.

Step 2: Alignment Scores

$e_{ti} = \mathbf{s}_t^T W_a \mathbf{h}_i$

Step 3: Attention Weights

$\alpha_{ti} = \frac{\exp(e_{ti})}{\sum_{j=1}^{T} \exp(e_{tj})}$

Step 4: Context Vector

$\mathbf{c}_t = \sum_{i=1}^{T} \alpha_{ti} \mathbf{h}_i$

Step 5: Attentional Hidden State

The key innovation—combine decoder state and context:

$\tilde{\mathbf{s}}_t = \tanh(W_c[\mathbf{s}_t; \mathbf{c}_t])$

WHERE $W_c \in \mathbb{R}^{d_s \times (d_s + d_h)}$ .

WHY? This learned combination decides how much to rely on context vs. decoder state. $\tanh$ bounds the output, preventing explosion.

Step 6: Output

$p(y_t | y_{<t}, \mathbf{x}) = \text{softmax}(W_o \tilde{\mathbf{s}}_t)$

Formula Luong Attention Complete (General)

\begin{align} \mathbf{s}_t &= \text{RNN}(\mathbf{y}_{t-1}, \mathbf{s}_{t-1}) \\ e_{ti} &= \mathbf{s}_t^T W_a \mathbf{h}_i \\ \alpha_{ti} &= \frac{\exp(e_{ti})}{\sum_{j=1}^{T} \exp(e_{tj})} \\ \mathbf{c}_t &= \sum_{i=1}^{T} \alpha_{ti} \mathbf{h}_i \\ \tilde{\mathbf{s}}_t &= \tanh(W_c[\mathbf{s}_t; \mathbf{c}_t]) \\ y_t &\sim \text{softmax}(W_o \tilde{\mathbf{s}}_t) \end{align}

Parameter count: $d_s \cdot d_h$ (for $W_a$ ) + $d_s(d_s + d_h)$ (for $W_c$ )

Side-by-Side Comparison

Aspect	Bahdanau (Additive)	Luong (Multiplicative)
When computed	Before decoder RNN step	After decoder RNN step
Uses decoder state	$\mathbf{s}_{t-1}$ (previous)	$\mathbf{s}_t$ (current)
Score function	$v_a^T \tanh(W_a \mathbf{s} + U_a \mathbf{h})$	$\mathbf{s}^T W_a \mathbf{h}$ (general)
Complexity	$O(T \cdot d_a \cdot (d_s + d_h))$	$O(T \cdot d_s \cdot d_h)$
Parameters	More (3 weight matrices)	Fewer (1-2 weight matrices)
Input feeding	Yes (context fed to RNN)	No (context combined after)
Speed	Slower	Faster
Performance	Slightly better on long sequences	Competitive, often preferred

WHY the timing difference matters:

Bahdanau: Context influences hidden state evolution → more integrated reasoning
Luong: Cleaner separation, easier to interpret and implement

Worked example Example 1: Translation Step

Task: Translate "The cat sat" → "Le chat s'est assis" Encoder states: $\mathbf{h}_1$ (The), $\mathbf{h}_2$ (cat), $\mathbf{h}_3$ (sat) Current output: Generating word2 (French: "chat")

Bahdanau Process:

Previous decoder state: $\mathbf{s}_1$ (after generating "Le")
Compute scores:

$e_{21} = v_a^T \tanh(W_a \mathbf{s}_1 + U_a \mathbf{h}_1) = 0.5$
$e_{22} = v_a^T \tanh(W_a \mathbf{s}_1 + U_a \mathbf{h}_2) = 2.1$ ← HIGH
$e_{23} = v_a^T \tanh(W_a \mathbf{s}_1 + U_a \mathbf{h}_3) = 0.3$

Softmax: $\alpha_{21} = 0.12, \alpha_{22} = 0.79, \alpha_{23} = 0.09$
Context: $\mathbf{c}_2 = 0.12\mathbf{h}_1 + 0.79\mathbf{h}_2 + 0.09\mathbf{h}_3$ (mostly "cat")
Update: $\mathbf{s}_2 = \text{GRU}([\text{emb}(\text{"Le"}); \mathbf{c}_2], \mathbf{s}_1)$
Output: "chat" with high probability

Why these steps? The model learns to attend to "cat" when generating "chat" through backprop. The alignment function discovers monotonic patterns in translation.

Worked example Example 2: Luong Attention Calculation

Dimension setup: $d_s = 256$ , $d_h = 512$ , $T = 5$ input tokens Step-by-step:

Decoder computes $\mathbf{s}_3 \in \mathbb{R}^{256}$ (after RNN step)
Score calculation (general):

$W_a \in \mathbb{R}^{256 \times 512}$
For each $i$ : $e_{3i} = \mathbf{s}_3^T W_a \mathbf{h}_i$ (256×1)ᵀ × (256×512) × (512×1) = scalar
Results: $[e_{31}, \ldots, e_{35}] = [1.2, 3.5, 0.8, 2.1, 0.5]$

Softmax: $\alpha_{32} = \frac{e^{3.5}}{e^{1.2}+e^{3.5}+e^{0.8}+e^{2.1}+e^{0.5}} = 0.64$ (attends mostly to position 2)
Context: $\mathbf{c}_3 = \sum_{i=1}^5 \alpha_{3i} \mathbf{h}_i \in \mathbb{R}^{512}$
Combine: $\tilde{\mathbf{s}}_3 = \tanh(W_c[\mathbf{s}_3; \mathbf{c}_3])$ where $[\mathbf{s}_3; \mathbf{c}_3] \in \mathbb{R}^{768}$
Final: $W_c \in \mathbb{R}^{256 \times 768}$ produces $\tilde{\mathbf{s}}_3 \in \mathbb{R}^{256}$ Why concatenate and project? The concatenation preserves both decoder reasoning and input context. The projection ( $W_c$ ) learns the optimal blend—maybe 70% decoder state, 30% context for this example.

Worked example Example 3: Computational Efficiency

Setup:

Input sequence length: $T = 50$
$d_s = d_h = 512$ , $d_a = 256$ (Bahdanau)
Batch size: 32 Bahdanau cost per decoder step:
Score computation: $T$ forward passes through alignment network
Each: $W_a \mathbf{s}_{t-1}$ : (256×512) × (512×1) = 131K ops
Plus: $U_a \mathbf{h}_i$ : (256×512) × (512×1) = 131K ops
Plus: $v_a^T \tanh(...)$ : 256 ops
Total per position: ~262K ops
For $T=50$ : 13.1M ops
Softmax: $50$ exponentials + normalization ≈ 150 ops
Context: weighted sum of 50 vectors ≈ 25K ops
Total: ~13.1M ops per decoder step

Luong cost per decoder step (general):

Score computation: $T$ bilinear products
$W_a \mathbf{h}_i$ : precompute once: 50× (512×512) = 13.1M ops
$\mathbf{s}_t^T (W_a \mathbf{h}_i)$ : 50 × 512 = 25K ops
Total: 13.1M (one-time) + 25K (per step)
Softmax: 150 ops
Context: 25K ops
Combine: $W_c[\mathbf{s}_t; \mathbf{c}_t]$ : (512×1024)×(1024×1) = 524K ops
Total: ~575K ops per step (after precomputation)

Why Luong is 20× faster: Matrix precomputation! $W_a \mathbf{h}_i$ is independent of $t$ , so compute ONCE for all decoder steps.

Common mistake Mistake 1: Forgetting to Mask Padding

Wrong approach: Apply softmax over all $T$ positions, including padding tokens.

# WRONG
scores = decoder_state @ encoder_states.T  # (T,)
attention_weights = softmax(scores)  # includes padding!

Why it feels right: Softmax normalizes anyway, so small scores on padding get small weights.

Why it's wrong: Even tiny attention on padding corrupts the context vector with meaningless information. For a sequence with40 real tokens and 10 padding, those 10 positions might get 0.01 weight each = 10% total mass wasted.

Fix: Mask padding positions with $-\infty$ before softmax:

# CORRECT
scores = decoder_state @ encoder_states.T
scores = scores.masked_fill(padding_mask, float('-inf'))
attention_weights = softmax(scores)  # padding gets exactly 0 weight

Why this works: $\exp(-\infty) = 0$ , so padding positions contribute nothing to the context.

Common mistake Mistake 2: Using Bahdanau Score for Luong Architecture

Wrong approach: Use additive scoring ( $v^T \tanh(...)$ ) but apply attention AFTER the decoder RNN step.

Why it feels right: "Bahdanau scoring is more expressive, so it should always be better."

The problem: Bahdanau's score function expects $\mathbf{s}_{t-1}$ (the state BEFORE seeing current input). When you use $\mathbf{s}_t$ (after seeing input), the alignment becomes less meaningful because $\mathbf{s}_t$ already "knows" what output it's generating.

Why this matters:

Bahdanau: "Given what I've generated so far ( $\mathbf{s}_{t-1}$ ), what input should I look at?"
Luong: "Given what my RNN thinks ( $\mathbf{s}_t$ ), how should I refine with input context?"

The timing of attention computation is coupled to the score function design.

Fix: Match scoring and timing:

Additive score → attention before RNN (Bahdanau)
Multiplicative score → attention after RNN (Luong)

Common mistake Mistake 3: Dimension Mismatch in Dot-Product Attention

Wrong setup:

encoder_dim = 512  # bidirectional LSTM: 256 forward + 256 backward
decoder_dim = 256  # unidirectional LSTM
 
score = decoder_state @ encoder_state.T  # ERROR: (256,) @ (512,)

Why it feels right: "Dot product is parameter-free and fast!"

Why it fails: Matrix dimension mismatch. Can't multiply $(1 \times 256)$ with $(1 \times 512)^T = (512 \times 1)$ .

Steel-man: You might think "I'll just use the first256 dims of encoder," but this discards half the information.

Fix 1: Use general/bilinear Luong:

W_a = nn.Linear(encoder_dim, decoder_dim)  # 512 → 256
score = decoder_state @ W_a(encoder_state).T  # Works!

Fix 2: Project decoder to encoder space:

W_d = nn.Linear(decoder_dim, encoder_dim)  # 256 → 512
score = W_d(decoder_state) @ encoder_state.T

Fix 3: Match dimensions from the start (make decoder512-dim).

Local vs Global Attention (Luong Extension)

Luong also introduced local attention to reduce complexity:

Global attention: Attend to ALL encoder positions (what we've described so far).

Complexity: $O(T \cdot d)$ per decoder step
Problem: Expensive for long sequences

Local attention: Attend to a WINDOW around predicted position.

Predict alignment position: $p_t = S \cdot \text{sigmoid}(v_p^T \tanh(W_p \mathbf{s}_t))$ where $S$ is source length
Define window: $[p_t - D, p_t + D]$ (e.g., $D=10$ )
Compute attention only within window
Apply Gaussian weighting: $\alpha_{ti} = \text{softmax}(e_{ti}) \cdot \exp\left(-\frac{(i-p_t)^2}{2\sigma^2}\right)$

Why Gaussian? Smoothly decay attention away from predicted center, preventing sharp cutoffs.

Result: Complexity drops from $O(T)$ to $O(D)$ per step, where $D \ll T$ .

Implementation Considerations

Batching:

# Encoder states: (batch, seq_len, hidden_dim)
# Decoder state: (batch, hidden_dim)
 
# Broadcasting for efficient score computation:
decoder_expanded = decoder_state.unsqueeze(1)  # (batch, 1, hidden_dim)
scores = (decoder_expanded @ encoder_states.transpose(1,2)).squeeze(1)
# Result: (batch, seq_len)

Numerical stability: Use log_softmax + exp instead of raw softmax for large $T$ :

log_alpha = torch.log_softmax(scores, dim=1)
alpha = torch.exp(log_alpha)

Teacher forcing: During training, use ground-truth $y_{t-1}$ even if model predicted something else. This stabilizes early training but can cause exposure bias at test time.

Recall Explain to a 12-year-old

Imagine you're translating a sentence from English to Spanish, word by word. The old way was like reading the entire English sentence once, closing the book, and trying to remember everything while you write in Spanish. If the sentence was long, you'd forget important parts!

Attention is like being allowed to keep the English book open. When you're writing each Spanish word, you can look back at the English sentence and focus on the part that's most helpful right now.

Bahdanau is like looking at the book BEFORE you start thinking about the next Spanish word. You check the English, then think.

Luong is like thinking first about what Spanish word might come next, THEN checking the English book to make sure you're on track.

Both work great! Luong is a bit faster because it can pre-read parts of the English book, while Bahdanau reads it fresh each time.

The "attention weights" are like highlighting: you highlight the English words that matter most for the current Spanish word. Sometimes one word is highlighted bright (90% attention), sometimes it's spread across few words.

Connections

3.5.1-Sequence-to-Sequence-Models: Both attention mechanisms extend seq2seq architecture
3.5.9-AttentionMechanism-Overview: These are specific implementations of general attention
3.5.11-Self-Attention-and-Transformers: Transformers evolved from these ideas, using self-attention
3.5.8-Encoder-Decoder-Architecture: Attention bridges encoder and decoder
2.4.7-Softmax-Function: Softmax converts scores to probability distribution
3.3.5-Vanishing-and-Exploding-Gradients: Attention helps by providing shorter gradient paths
4.2.3-Beam-Search: Attention weights guide beam search in decoding
5.1.2-Word-Embeddings: Attention operates in embedding space

#flashcards/ai-ml

What is the key difference between Bahdanau and Luong attention timing? :: Bahdanau computes attention BEFORE the decoder RNN step (uses $\mathbf{s}_{t-1}$ ), while Luong computes attention AFTER the decoder RNN step (uses $\mathbf{s}_t$ ).

What is the Bahdanau attention score function?

e_{ti} = v_a^T \tanh(W_a \mathbf{s}_{t-1} + U_a \mathbf{h}_i)

— an additive/concat mechanism using a feedforward network with tanh nonlinearity.

What is the Luong general attention score function?

e_{ti} = \mathbf{s}_t^T W_a \mathbf{h}_i

— a bilinear/multiplicative mechanism that learns to map encoder space to decoder space.

What is input feeding in Bahdanau attention?

The context vector

\mathbf{c}_t

is concatenated with the input embedding and fed to the decoder RNN:

\mathbf{s}_t = \text{RNN}([\mathbf{y}_{t-1}; \mathbf{c}_t], \mathbf{s}_{t-1})

. This allows attention to directly influence hidden state evolution.

How does Luong attention combine context with decoder state?

It creates an attentional hidden state:

\tilde{\mathbf{s}}_t = \tanh(W_c[\mathbf{s}_t; \mathbf{c}_t])

which combines the decoder state and context vector through a learned projection.

What are the three Luong attention scoring variants?

1) Dot:

\mathbf{s}_t^T \mathbf{h}_i

(no parameters, requires same dims), 2) General:

\mathbf{s}_t^T W_a \mathbf{h}_i

(most common), 3) Concat:

v_a^T \tanh(W_a[\mathbf{s}_t; \mathbf{h}_i])

(similar to Bahdanau).

Why is Luong attention typically faster than Bahdanau?

Luong can precompute

W_a \mathbf{h}_i

for all encoder positions once, since it's independent of the decoder state. Bahdanau must compute the additive function separately for each encoder position at every decoder step.

What is the formula for attention weights from alignment scores?

\alpha_{ti} = \frac{\exp(e_{ti})}{\sum_{j=1}^{T} \exp(e_{tj})}

— softmax normalization to create a probability distribution that sums to 1.

What is the context vector formula in both mechanisms?

\mathbf{c}_t = \sum_{i=1}^{T} \alpha_{ti} \mathbf{h}_i

— a weighted sum of all encoder hidden states using attention weights.

Why must padding positions be masked before computing attention?

Unmasked padding gets small but non-zero attention weights, wasting probability mass and corrupting the context vector with meaningless information. Masking with

-\infty

before softmax ensures padding positions get exactly 0 weight.

What is local attention (Luong extension)?

An optimization that attends to only a window

[p_t - D, p_t + D]

around a predicted alignment position

p_t

, reducing complexity from

O(T)

O(D)

per decoder step.

What is the key architectural motivation for attention mechanisms?

To solve the fixed-length bottleneck problem in seq2seq models — instead of compressing the entire input into one vector, attention allows the decoder to dynamically access all encoder states.

How many weight matrices does Bahdanau attention require?

Three:

W_a \in \mathbb{R}^{d_a \times d_s}

for decoder projection,

U_a \in \mathbb{R}^{d_a \times d_h}

for encoder projection, and

v_a \in \mathbb{R}^{d_a}

for the alignment vector. Total parameters:

d_a(d_s + d_h +1)

How many weight matrices does Luong general attention require?

Two:

W_a \in \mathbb{R}^{d_s \times d_h}

for scoring (parameters:

d_s \cdot d_h

) and

W_c \in \mathbb{R}^{d_s \times (d_s+d_h)}

for combining context and decoder state (parameters:

d_s(d_s+d_h)

Why does Bahdanau use tanh nonlinearity in the score function?

To allow the alignment function to learn complex, non-linear relationships between decoder and encoder states. Without nonlinearity, it would collapse to a simple linear transformation.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Chalo ek simple baat se shuru karte hain. Socho tumhe pura ek book padhkar uska sirf ek chhoti si line mein summary banana hai, aur phir usi ek line ke basis pe review likhna hai. Bahut saari important details toh gayab ho jayengi na? Yahi problem thi purane seq2seq models mein — woh pure input sequence ko ek fixed-size vector mein daba dete the, aur long sentences ke liye yeh "bottleneck" bahut information loss karta tha. Attention ka core idea yahi hai ki decoder ko pura book khula rakhne do — har output word banate waqt woh saare encoder states ko dekh sake aur dynamically decide kare ki abhi kaun sa part sabse relevant hai. Bilkul jaise review likhte waqt tum zaroorat ke hisaab se pages palat lo.

Ab Bahdanau aur Luong attention dono isi kaam ko karte hain, bas farak yeh hai ki alignment score kaise calculate karte hain. Bahdanau (additive attention) ek chhota feedforward network use karta hai — decoder state aur encoder state dono ko ek common space mein project karta hai, tanh se non-linearity add karta hai, phir score nikaalta hai. Iska fayda yeh hai ki model khud "seekh" leta hai ki alignment kaise karna hai. Luong (multiplicative) simple dot product use karta hai, jo maanke chalta hai ki spaces already aligned hain. Dono cases mein score nikalne ke baad softmax lagta hai (taaki weights probability ban jaayein aur sum 1 ho), phir un weights se encoder states ka weighted sum lete hain — yeh banta hai context vector, jo current word predict karne ke liye relevant summary hota hai.

Yeh samajhna kyun zaroori hai? Kyunki attention modern AI ki backbone hai — Google Translate se lekar ChatGPT tak, sab jagah yahi concept scale hua hai. Transformers, jo aaj sabse powerful models hain, seedhe isi attention idea ka advanced version hain. Toh agar tum yeh core intuition — "fixed bottleneck se bacho, dynamically focus karo" — aur alignment score ka basic maths samajh loge, toh tumhare paas poore modern deep learning ki foundation aa jayegi. Chhote se concept se badi cheezein banti hain, bas basics clear hone chahiye!

What Are Bahdanau and Luong Attention?

The Shared Framework

Bahdanau Attention (Additive/Concat)

Derivation from First Principles

Luong Attention (Multiplicative)

Three Scoring Variants

Complete Luong Attention (General)

Side-by-Side Comparison

Local vs Global Attention (Luong Extension)

Implementation Considerations

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Sequence Models

Connections