The original - Attention is All You Need - architecture

What Problem Did It Solve?

Traditional seq2seq models had three pain points:

Sequential bottleneck: RNNs process token-by-token, can't parallelize across sequence
Long-range dependencies: Information degrades over many steps (vanishing gradients)
Fixed context window: CNs have limited receptive fields

The AIAYN architecture eliminated recurrence and convolution entirely, using only attention mechanisms and feed-forward networks.

The Complete Architecture

Architecture Components (Detailed Breakdown)

1. Input Embedding + Positional Encoding

WHY: Raw tokens are discrete symbols. We need continuous vectors that capture both meaning (embedding) and position (since attention has no inherent order).

Formula Derivation: $\text{InputRepresentation}(x, \text{pos}) = \text{Embed}(x) + \text{PE}(\text{pos})$

The positional encoding uses sine/cosine functions: $\text{PE}(\text{pos}, 2i) = \sin\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$ $\text{PE}(\text{pos}, 2i+1) = \cos\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$

WHY these specific functions?

Sinusoids allow the model to learn relative positions: $\text{PE}(\text{pos}+k)$ can be expressed as a linear function of $\text{PE}(\text{pos})$
Different frequencies (via $10000^{2i/d_{\text{model}}}$ ) encode position at different scales
Extrapolates to sequence lengths unseen during training

Worked example Position Encoding in Action

For $d_{\text{model}} = 4$ , position 3:

$\text{PE}(3, 0) = \sin(3 / 10000^{0/4}) = \sin(3) = 0.141$
$\text{PE}(3, 1) = \cos(3 / 10000^{0/4}) = \cos(3) = -0.990$
$\text{PE}(3, 2) = \sin(3 / 10000^{2/4}) = \sin(3/100) = 0.030$
$\text{PE}(3, 3) = \cos(3 / 10000^{2/4}) = \cos(3/100) = 0.9995$

Note $10000^{2/4} = 10000^{0.5} = 100$ , so the argument is $3/100 = 0.03$ . Higher frequency (dimensions 0,1) changes rapidly with position. Lower frequency (dimensions 2,3) changes slowly, encoding "coarse" position.

2. Encoder Layer (×6 Stack)

Each of the 6 encoder layers contains:

Sub-layer 1: Multi-Head Self-Attention

The attention mechanism computes: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

Derivation from scratch:

Start with input $X \in \mathbb{R}^{n \times d_{\text{model}}}$ (n tokens, $d_{\text{model}}=512$ dims)
Project to queries/keys/values: $Q = XW^Q$ , $K = XW^K$ , $V = XW^V$ where $W^Q, W^K, W^V \in \mathbb{R}^{d_{\text{model}} \times d_k}$
Compute similarity scores: $QK^T \in \mathbb{R}^{n \times n}$ (every token attends to every token)
Scale by $\sqrt{d_k}$ WHY? Dot products grow with dimension; large values push softmax into saturation regions with tiny gradients
Apply softmax row-wise: attention weights sum to 1 per query
Weighted sum of values: output per token is a weighted combination of all tokens' value vectors

Multi-head attention runs $h=8$ attention operations in parallel: $\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O$ $\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$

Each head has $d_k = d_v = d_{\text{model}}/h = 512/8 = 64$ dimensions.

WHY multiple heads? Different learn different types of relationships (syntactic, semantic, positional). One head might focus on subject-verb agreement, another on long-range dependencies.

Worked example Attention Computation Walkthrough

Input sentence: "The cat sat" (3 tokens), $d_{\text{model}}=4$ , single head, $d_k=4$

Step 1: Embedings (simplified): $X = \begin{bmatrix} 0.1 & 0.2 & 0.3 & 0.4 \\ 0.5 & 0.6 & 0.7 & 0.8 \\ 0.2 & 0.3 & 0.4 & 0.5 \end{bmatrix}$

Step 2: Projection matrices (random init): $W^Q = W^K = W^V = I_4 \text{ (identity for simplicity)}$ Thus $Q = K = V = X$

Step 3: Compute scores $QK^T$ (row $i$ , col $j$ = dot product of row $i$ and row $j$ of $X$ ):

$X_1 \cdot X_1 = 0.01+0.04+0.09+0.16 = 0.30$
$X_1 \cdot X_2 = 0.05+0.12+0.21+0.32 = 0.70$
$X_1 \cdot X_3 = 0.02+0.06+0.12+0.20 = 0.40$
$X_2 \cdot X_2 = 0.25+0.36+0.49+0.64 = 1.74$
$X_2 \cdot X_3 = 0.10+0.18+0.28+0.40 = 0.96$
$X_3 \cdot X_3 = 0.04+0.09+0.16+0.25 = 0.54$ $QK^T = \begin{bmatrix} 0.30 & 0.70 & 0.40 \\ 0.70 & 1.74 & 0.96 \\ 0.40 & 0.96 & 0.54 \end{bmatrix}$ Why this step? Measures how much each token should attend to each other token via dot product similarity.

Step 4: Scale by $\sqrt{d_k} = 2$ : $\frac{QK^T}{\sqrt{d_k}} = \begin{bmatrix} 0.15 & 0.35 & 0.20 \\ 0.35 & 0.87 & 0.48 \\ 0.20 & 0.48 & 0.27 \end{bmatrix}$ Why this step? Prevents extreme values that would make softmax derivatives vanish.

Step 5: Softmax per row (attention weights). For row 1, $e^{0.15}=1.162$ , $e^{0.35}=1.419$ , $e^{0.20}=1.221$ , sum $=3.802$ : $\text{softmax}(\text{row 1}) = [0.306, 0.373, 0.321]$ Why this step? Normalizes attention scores to valid probability distribution.

Step 6: Weighted sum $\times V$ : For token 1: $0.306 \cdot V_1 + 0.373 \cdot V_2 + 0.321 \cdot V_3$ Output captures information from all tokens weighted by relevance!

Sub-layer 2: Position-wise Feed-Forward Network

$\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$

Where $W_1 \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ , $W_2 \in \mathbb{R}^{d_{ff} \times d_{\text{model}}}$ , $d_{ff} = 2048$ .

WHY? Attention is linear (weighted sums). FFN adds non-linearity and increases model capacity. Applied to each position independently (same network across positions, different positions don't interact here).

Residual Connection + Layer Norm around each sub-layer: $\text{LayerNorm}(x + \text{Sublayer}(x))$

WHY residuals? Enable gradient flow in deep networks (6 layers = 12 sub-layers in encoder). Without residuals, gradients vanish.

WHY LayerNorm? Stabilizes training by normalizing across features per example: $\text{LayerNorm}(x) = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \cdot \gamma + \beta$ where $\mu, \sigma^2$ computed over $d_{\text{model}}$ dimensions per token.

3. Decoder Layer (×6 Stack)

Each decoder layer has three sub-layers:

Sub-layer 1: Masked Multi-Head Self-Attention

Same as encoder self-attention BUT with causal masking: $\text{mask}_{ij} = \begin{cases} 0 & \text{if } i \geq j \\ -\infty & \text{if } i < j \end{cases}$

Applied before softmax: $\text{softmax}(QK^T/\sqrt{d_k} + \text{mask})$

WHY? During training, decoder sees the full target sequence. Masking prevents position $i$ from attending to future positions $j > i$ , ensuring autoregressive generation (predict next token using only past tokens).

Sub-layer 2: Encoder-Decoder Attention (Cross-Attention)

$Q = \text{DecoderOutput}, \quad K = V = \text{EncoderOutput}$

WHY? This is where decoder accesses source sequence! Decoder queries encoder's representation of input. For translation "Le chat" → "The cat", decoder generating "cat" attends to "chat" in encoder output.

Sub-layer 3: Position-wise FFN (same as encoder)

Residual + LayerNorm around all three sub-layers.

4. Final Linear + Softmax

$P(\text{word}) = \text{softmax}(\text{DecoderOutput} \cdot W_{\text{vocab}} + b)$

Projects $d_{\text{model}}=512$ to vocabulary size (e.g., 37000), producing probability distribution over next token.

Training Details from Paper

Formula Training Hyperparameters (Base Model)

$d_{\text{model}} = 512$ (embedding/hidden dimension)
$h = 8$ attention heads
$d_k = d_v = 64$ per head
$d_{ff} = 2048$ (FFN inner dimension)
$N = 6$ layers encoder and decoder
Dropout rate: 0.1 (applied to sub-layer outputs, attention weights, embedings)
Label smoothing: $\epsilon_{ls} = 0.1$ WHY label smoothing? Instead of one-hot targets $[0,1,0,0]$ , distribute mass: the true class gets $1-\epsilon_{ls}+\epsilon_{ls}/K$ and each other class gets $\epsilon_{ls}/K$ . For $K=4$ , true class $= 0.9 + 0.025 = 0.925$ , others $= 0.025$ each. Prevents overconfident predictions, improves generalization.

Optimizer: Adam with $\beta_1=0.9, \beta_2=0.98, \epsilon=10^{-9}$

Learning Rate Schedule: $lr = d_{\text{model}}^{-0.5} \cdot \min(\text{step}^{-0.5}, \text{step} \cdot \text{warmup\_steps}^{-1.5})$ with warmup_steps = 4000.

WHY this schedule?

Linear warmup: prevents instability from large gradients early on
Inverse square root decay: gradually reduces LR as model converges
Scaling by $d_{\text{model}}^{-0.5}$ : normalizes for model size

Why This Architecture Dominates

Parallelization: All positions processed simultaneously (vs. sequential RNNs)
Constant path length: Any token pair directly connected via attention (vs. $O(n)$ steps in RNN)
Learnable relationships: Attention weights adapt to data (vs. fixed CNN kernels)
Scalability: Stack more layers/heads/dimensions as compute grows

Recall Explain to a 12-Year-Old

Imagine you're reading a long story. Your brain doesn't read one word, forget it, then read the next. Instead, your eyes can jump around—you might read a word, then remember something from the beginning, then connect it to the end. That's what Transformers do!

Old AI models (RNNs) were like reading with a tiny window that slides along: see one word, move to the next, but forget the first. Transformers instead see the WHOLE sentence at once. Every word looks at every other word and decides "How much should I pay attention to that word to understand this one?"

There are two parts: an Encoder (reads the input sentence, like "Le chat noir" in French) and a Decoder (writes the output, like "The black cat"). The Encoder figures out what everything means. The Decoder creates the answer, but with a special trick: when writing word3, it can only look at words 1 and 2 (no peeking at future words!), just like you can't see tomorrow when writing your diary today.

The magic ingredient is attention—a formula that computes "How related are word A and word B?" for every pair. Do this millions of times while training, and the AI learns language patterns!

Connections

Multi-Head Self-Attention — core mechanism used in both encoder and decoder
Positional Encoding — how Transformers inject sequence order
Scaled Dot-Product Attention — the fundamental attention formula
Encoder-Decoder Architecture — general seq2seq framework
Residual Connections — gradient flow technique from ResNet
Layer Normalization — stabilization technique
Teacher Forcing — training technique for autoregressive decoders
Attention Masking — causal masking in decoder
Tokenization — converting text to input IDs
Beam Search — decoding strategy at inference

Flashcards

#flashcards/ai-ml

What are the three sub-layers in each Transformer decoder layer?

(1) Masked multi-head self-attention, (2) Encoder-decoder cross-attention, (3) Position-wise feed-forward network. Each has residual connection + LayerNorm.

Why does the attention formula use scaling by √d_k?

Dot products grow in magnitude with dimension d_k. Large values push softmax into saturation regions where gradients are extremely small, hindering learning. Dividing by √d_k keeps variance stable.

What is the purpose of causal masking in the decoder?

Prevents position i from attending to future positions j > i during training. This enforces autoregressive generation: predicting token i can only use tokens 1 through i-1, not future tokens.

How many parameters does the Transformer base model have approximately?

~65 million parameters, using weight tying between input embedding, output embedding, and pre-softmax projection.

What is the learning rate warmup schedule used in the original paper?

lr = d_model^(-0.5) × min(step^(-0.5), step × warmup_steps^(-1.5)) with warmup_steps=4000. Linear increase for 4000 steps, then inverse square root decay.

Why multiply embedings by √d_model before adding positional encoding?

Embedings have variance ~1, positional encodings are bounded [-1,1]. Scaling by √d_model (~22.6 for d=512) ensures embedings dominate initially while PE adds positional information without overwhelming semantic content.

What is the dimension of each attention head in the base model?

d_k = d_v = d_model / h = 512 / 8 = 64 dimensions per head.

Why use sinusoidal positional encoding instead of learned embedings?

Sinusoids allow the model to extrapolate to sequence lengths longer than seen during training. PE(pos+k) can be expressed as a linear function of PE(pos), helping the model learn relative positions.

What is the FFN inner dimension d_ff in the base model?

2048 (4× the model dimension of 512). Projects up, applies ReLU, projects back down.

How does encoder-decoder attention differ from self-attention?

In cross-attention, Q comes from decoder, but K and V come from encoder output. This allows decoder to attend to source sequence while generating target sequence.

With label smoothing ε_ls=0.1 and vocabulary size K, what target probability does the true class get?

1 − ε_ls + ε_ls/K. For K=4 this is 0.9 + 0.025 = 0.925; every other class gets ε_ls/K = 0.025.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Chalo, is note ka core idea samajhte hain. 2017 se pehle, jab bhi humein language ya kisi bhi sequence ke saath kaam karna hota tha, hum RNNs ya LSTMs use karte the. Problem yeh thi ki yeh models ek-ek word ko left-to-right process karte the, matlab ek word tab tak process nahi hota jab tak pichla word finish na ho. Yeh slow tha aur parallel processing possible hi nahi thi. Doosri badi dikkat thi long-range dependencies ki — agar sentence bada hota, toh shuruaat ki information end tak pahunchte-pahunchte fade ho jaati thi (vanishing gradient ki wajah se). Transformer ne bola: "Kyun na hum saare words ko ek saath process karein, aur har word ko baaki sabhi words ko directly dekhne ki azaadi dein?" Isko hi self-attention kehte hain, aur yahi poori revolution ki jaan hai.

Ab yahan ek interesting twist hai. Kyunki hum saare words ek saath dekh rahe hain, model ko yeh pata hi nahi chalta ki kaunsa word pehle aaya aur kaunsa baad mein — order ki information gaayab ho jaati hai. Isko solve karne ke liye positional encoding add karte hain, jo sine aur cosine functions use karti hai taaki har position ka ek unique "signature" ban jaaye. Yeh smartly design kiya gaya hai — different frequencies alag-alag scale par position batati hain, aur model relative positions bhi seekh sakta hai. Ek chhoti si detail: embeddings ko $\sqrt{d_{\text{model}}}$ se multiply karte hain taaki meaning (semantic content) shuruaati training mein dominate kare aur position ki info usse overwhelm na kar de.

Yeh sab kyun matter karta hai? Kyunki aaj jitne bhi bade models hain — ChatGPT, BERT, translation systems — sab isi "Attention is All You Need" architecture par khade hain. Yeh design GPUs par massively parallel chal sakta hai, isliye hum huge datasets par train kar paate hain jo pehle impossible tha. Encoder-decoder structure, multi-head attention, residual connections aur layer normalization — yeh saare pieces milkar ek aisa foundation banate hain jo modern AI ki backbone hai. Toh agar tum AI-ML seriously seekhna chahte ho, yeh architecture samajhna non-negotiable hai — yahi se sab kuch shuru hota hai.

The original - Attention is All You Need - architecture

What Problem Did It Solve?

The Complete Architecture

Architecture Components (Detailed Breakdown)

1. Input Embedding + Positional Encoding

2. Encoder Layer (×6 Stack)

3. Decoder Layer (×6 Stack)

4. Final Linear + Softmax

Training Details from Paper

Why This Architecture Dominates

Connections

Flashcards

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Transformer Architecture

Connections