Multi-head attention

What Problem Does It Solve?

Single-head attention has limited representational capacity. A single set of $Q, K, V$ projections can only learn one type of relationship pattern. But language (and most sequential data) has multiple simultaneous structure types:

• Syntactic dependencies (subject agrees with verb)
• Semantic relationships (synonym, antonyms)
• Positional patterns (nearby words vs. distant context)
• Co-reference (pronouns linking to earlier nouns)

Multi-head attention (MHA) learns multiple representation subspaces in parallel, each head specializing in different patterns.

Definition and Architecture

Derivation From First Principles

Step 1: Why Reduce Dimension Per Head?

Goal: Run $h$ attention heads in parallel without increasing computational cost.

If we have $d_{\text{model}} = 512$ and want $h = 8$ heads:

• Naive approach: Each head uses full512-dim → total cost $8 \times \text{cost}(512)$
• Smart approach: Each head uses $d_k = 512/8 = 64$ → total cost $\approx \text{cost}(512)$

Why this works: Attention cost is $O(n^2 d)$ where $n$ = sequence length, $d$ = dimension.

• 8 heads of 64-dim each: $8 \times O(n^2 \times 64) = O(n^2 \times 512)$
• Same total FLOPs as one512-dim head, but we get8 different views!

Step 2: Projecting Into Head Subspaces

For head $i$, we project the shared $Q, K, V$ (shape $[n, d_{\text{model}}]$) down to $[n, d_k]$:

$Q_i = Q W_i^Q, \quad K_i = K W_i^K, \quad V_i = V W_i^V$

where $W_i^Q, W_i^K, W_i^V$ are learned, different for each head.

WHY different projections?

• If all heads used the same $W^Q, W^K, W^V$, they'd compute identical attention patterns—no benefit!
• Different projections let each head learn to extract different features from the same input.

Intuition: Think of each $W_i^Q$ as asking different question:

• Head 1 might project to capture "What words are grammatically related?"
• Head 2 might project to capture "What words are semantically similar?"
• Head 3 might project to capture "What words are nearby in position?"

Step 3: Compute Scaled Dot-Product Attention Per Head

Each head independently computes:

$\text{head}_i = \text{softmax}\left(\frac{Q_i K_i^T}{\sqrt{d_k}}\right) V_i$

This gives an output of shape $[n, d_v]$ for each head.

WHY the same scaled dot-product formula?

• It works! The magic is in the different learned projections $W_i^{Q,K,V}$, not in the attention mechanism itself.
• Using the same mechanism means each head is interpretable in the same way.

Step 4: Concatenate Heads

Stack the $h$ head outputs side-by-side:

$\text{Concat}(\text{head}_1, \ldots, \text{head}_h) = [\text{head}_1 \mid \text{head}_2 \mid \cdots \mid \text{head}_h]$

This produces shape $[n, h \cdot d_v] = [n, d_{\text{model}}]$ (since $h \cdot d_v = d_{\text{model}}$).

WHY concatenate instead of sum/average?

• We want to preserve all information from every head.
• Suming would merge them, losing the distinct patterns each head learned.
• Concatenation keeps them separate, but we need one more step to blend them...

Step 5: Final Linear Projection

$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O$

where $W^O \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$ is learned.

WHY this final projection?

• The concatenated heads are still in separate "channels"—$W^O$ lets the model mix information across heads.
• It learns how to combine different types of attention (syntactic + semantic + positional).
• Returns output to full $d_{\text{model}}$ dimension, ready for the next layer.

Complete Forward Pass Example

Common Mistakes

Parameter Count

Let's count parameters for multi-head attention:

Per head:

• $W_i^Q$: $d_{\text{model}} \times d_k$
• $W_i^K$: $d_{\text{model}} \times d_k$
• $W_i^V$: $d_{\text{model}} \times d_v$
• Total per head: $3 d_{\text{model}} \cdot d_k$

All heads: $h \cdot 3 d_{\text{model}} \cdot d_k = 3 d_{\text{model}} \cdot (h \cdot d_k) = 3 d_{\text{model}}^2$ (since $h \cdot d_k = d_{\text{model}}$)

Output projection: $W^O$ has $d_{\text{model}} \times d_{\text{model}} = d_{\text{model}}^2$ parameters

Total: $3 d_{\text{model}}^2 + d_{\text{model}}^2 ==4 d_{\text{model}}^2==$ parameters

Computational Complexity

Per head:

• Computing $Q_i, K_i, V_i$: $O(n \cdot d_{\text{model}} \cdot d_k)$ each, total $3 n \cdot d_{\text{model}} \cdot d_k$
• Attention scores $Q_i K_i^T$: $O(n^2 d_k)$
• Softmax: $O(n^2)$
• Weighted sum with $V_i$: $O(n^2 d_v)$

All heads: $h$ times the per-head cost

Dominant term: Attention scores $O(h \cdot n^2 d_k) = O(n^2 d_{\text{model}})$ (since $h \cdot d_k = d_{\text{model}}$)

Output projection: $O(n \cdot d_{\text{model}}^2)$

For long sequences where $n \gg d_{\text{model}}$, the $O(n^2 d_{\text{model}})$ attention term dominates. This is why Transformers struggle with very long sequences (length 10,000+)—quadratic in sequence length!

Connections Scaled Dot-Product Attention: The attention mechanism used within each head

• Self-Attention: MHA is typically used in self-attention (Q, K, V from same source)
• Cross-Attention: MHA can also be used for encoder-decoder attention (Q from decoder, K,V from encoder)
• Transformer Encoder Layer: Uses MHA followed by feed-forward network
• Positional Encoding: Provides position information that attention heads can leverage
• Attention Visualization: Techniques to interpret what each head has learned
• Query-Key-Value Intuition: Foundation for understanding the projections
• Linear Projections in Neural Networks: What the $W^Q, W^K, W^V W^O$ matrices do

#flashcards/ai-ml

What is the primary advantage of multi-head attention over single-head attention? :: Multi-head attention learns multiple representation subspaces in parallel, allowing the model to capture different types of relationships (syntactic, semantic, positional) simultaneously. A single head has limited representational capacity.

Concept Map

Hinglish (regional understanding)