The original - Attention is All You Need - architecture

4.1.12 · AI-ML › Transformer Architecture

What Problem Did It Solve?

Traditional seq2seq models mein teen pain points the:

Sequential bottleneck: RNNs token-by-token process karte hain, sequence ke across parallelize nahi kar sakte
Long-range dependencies: Information kaafi steps mein degrade ho jaati hai (vanishing gradients)
Fixed context window: CNNs ke limited receptive fields hote hain

AIAYN architecture ne recurrence aur convolution ko bilkul eliminate kar diya, sirf attention mechanisms aur feed-forward networks use karke.

The Complete Architecture

Architecture Components (Detailed Breakdown)

1. Input Embedding + Positional Encoding

WHY: Raw tokens discrete symbols hote hain. Humein continuous vectors chahiye jo dono meaning (embedding) aur position (kyunki attention mein inherently koi order nahi hota) capture karein.

Formula Derivation: $InputRepresentation (x, pos) = Embed (x) + PE (pos)$

Positional encoding sine/cosine functions use karta hai: $PE (pos, 2 i) = sin (\frac{pos}{1000 0 ^{2 i / d_{model}}})$ $PE (pos, 2 i + 1) = cos (\frac{pos}{1000 0 ^{2 i / d_{model}}})$

WHY ye specific functions?

Sinusoids model ko relative positions seekhne dete hain: $PE (pos + k)$ ko $PE (pos)$ ke linear function ke roop mein express kiya ja sakta hai
Alag-alag frequencies ( $1000 0^{2 i / d_{model}}$ ke through) alag-alag scales par position encode karti hain
Training ke dauran unseen sequence lengths tak extrapolate karta hai

2. Encoder Layer (×6 Stack)

6 encoder layers mein se har ek mein ye hota hai:

Sub-layer 1: Multi-Head Self-Attention

Attention mechanism compute karta hai: $Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V$

Scratch se Derivation:

Input se shuru karo $X \in R^{n \times d_{model}}$ (n tokens, $d_{model} = 512$ dims)
Queries/keys/values mein project karo: $Q = X W^{Q}$ , $K = X W^{K}$ , $V = X W^{V}$ jahan $W^{Q}, W^{K}, W^{V} \in R^{d_{model} \times d_{k}}$
Similarity scores compute karo: $Q K^{T} \in R^{n \times n}$ (har token har token ko attend karta hai)
$d_{k}$ se scale karo WHY? Dot products dimension ke saath badhte hain; bade values softmax ko saturation regions mein push karte hain jahan gradients tiny hote hain
Row-wise softmax apply karo: attention weights har query ke liye 1 tak sum hote hain
Values ka weighted sum: har token ka output sabhi tokens ke value vectors ka weighted combination hota hai

Multi-head attention $h = 8$ attention operations parallel mein run karta hai: $MultiHead (Q, K, V) = Concat (head_{1}, \dots, head_{h}) W^{O}$ $head_{i} = Attention (Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V})$

Har head ka $d_{k} = d_{v} = d_{model} / h = 512/8 = 64$ dimensions hote hain.

WHY multiple heads? Alag-alag heads alag-alag types ke relationships (syntactic, semantic, positional) seekhte hain. Ek head subject-verb agreement par focus kar sakta hai, doosra long-range dependencies par.

Worked example Attention Computation Walkthrough

Input sentence: "The cat sat" (3 tokens), $d_{model} = 4$ , single head, $d_{k} = 4$

Step 1: Embeddings (simplified): $X = 0.1 0.5 0.2 0.2 0.6 0.3 0.3 0.7 0.4 0.4 0.8 0.5$

Step 2: Projection matrices (random init): $W^{Q} = W^{K} = W^{V} = I_{4} (identity for simplicity)$ Thus $Q = K = V = X$

Step 3: Scores compute karo $Q K^{T}$ (row $i$ , col $j$ = $X$ ki row $i$ aur row $j$ ka dot product):

$X_{1} \cdot X_{1} = 0.01 + 0.04 + 0.09 + 0.16 = 0.30$
$X_{1} \cdot X_{2} = 0.05 + 0.12 + 0.21 + 0.32 = 0.70$
$X_{1} \cdot X_{3} = 0.02 + 0.06 + 0.12 + 0.20 = 0.40$
$X_{2} \cdot X_{2} = 0.25 + 0.36 + 0.49 + 0.64 = 1.74$
$X_{2} \cdot X_{3} = 0.10 + 0.18 + 0.28 + 0.40 = 0.96$
$X_{3} \cdot X_{3} = 0.04 + 0.09 + 0.16 + 0.25 = 0.54$ $Q K^{T} = 0.30 0.70 0.40 0.70 1.74 0.96 0.40 0.96 0.54$ Ye step kyun? Measure karta hai ki har token ko dot product similarity ke through kisi doosre token ko kitna attend karna chahiye.

Step 4: $d_{k} = 2$ se scale karo: $\frac{Q K ^{T}}{d _{k}} = 0.15 0.35 0.20 0.35 0.87 0.48 0.20 0.48 0.27$ Ye step kyun? Extreme values prevent karta hai jo softmax derivatives ko vanish kara de.

Step 5: Softmax per row (attention weights). Row 1 ke liye, $e^{0.15} = 1.162$ , $e^{0.35} = 1.419$ , $e^{0.20} = 1.221$ , sum $= 3.802$ : $softmax (row 1) = [0.306, 0.373, 0.321]$ Ye step kyun? Attention scores ko valid probability distribution mein normalize karta hai.

Step 6: Weighted sum $\times V$ : Token 1 ke liye: $0.306 \cdot V_{1} + 0.373 \cdot V_{2} + 0.321 \cdot V_{3}$ Output sabhi tokens ki information relevance ke basis par weighted karke capture karta hai!

Sub-layer 2: Position-wise Feed-Forward Network

$FFN (x) = ReLU (x W_{1} + b_{1}) W_{2} + b_{2}$

Jahan $W_{1} \in R^{d_{model} \times d_{f f}}$ , $W_{2} \in R^{d_{f f} \times d_{model}}$ , $d_{f f} = 2048$ .

WHY? Attention linear hai (weighted sums). FFN non-linearity add karta hai aur model capacity badhata hai. Har position par independently apply hota hai (same network positions ke across, alag positions yahan interact nahi karte).

Har sub-layer ke around Residual Connection + Layer Norm: $LayerNorm (x + Sublayer (x))$

WHY residuals? Deep networks (6 layers = encoder mein 12 sub-layers) mein gradient flow enable karte hain. Residuals ke bina, gradients vanish ho jaate hain.

WHY LayerNorm? Training stabilize karta hai har example ke liye features ke across normalize karke: $LayerNorm (x) = \frac{x - μ}{σ ^{2} + ϵ} \cdot γ + β$ jahan $μ, σ^{2}$ har token ke liye $d_{model}$ dimensions ke over compute hote hain.

3. Decoder Layer (×6 Stack)

Har decoder layer mein teen sub-layers hain:

Sub-layer 1: Masked Multi-Head Self-Attention

Encoder self-attention jaisa hi lekin causal masking ke saath: $mask_{ij} = {0 - \infty if i \geq j if i < j$

Softmax se pehle apply hota hai: $softmax (Q K^{T} / d_{k} + mask)$

WHY? Training ke dauran, decoder puri target sequence dekhta hai. Masking position $i$ ko future positions $j > i$ attend karne se rokta hai, autoregressive generation ensure karta hai (sirf past tokens use karke next token predict karo).

Sub-layer 2: Encoder-Decoder Attention (Cross-Attention)

$Q = DecoderOutput, K = V = EncoderOutput$

WHY? Yahan decoder source sequence access karta hai! Decoder encoder ki input representation ko query karta hai. Translation "Le chat" → "The cat" ke liye, "cat" generate karte waqt decoder encoder output mein "chat" ko attend karta hai.

Sub-layer 3: Position-wise FFN (encoder jaisa hi)

Teeno sub-layers ke around Residual + LayerNorm.

4. Final Linear + Softmax

$P (word) = softmax (DecoderOutput \cdot W_{vocab} + b)$

$d_{model} = 512$ ko vocabulary size (e.g., 37000) tak project karta hai, next token par probability distribution produce karta hai.

Training Details from Paper

Optimizer: Adam with $β_{1} = 0.9, β_{2} = 0.98, ϵ = 1 0^{- 9}$

Learning Rate Schedule: $l r = d_{model}^{- 0.5} \cdot min (step^{- 0.5}, step \cdot warmup_steps^{- 1.5})$ with warmup_steps = 4000.

WHY ye schedule?

Linear warmup: early mein large gradients se instability prevent karta hai
Inverse square root decay: model converge hone par LR gradually reduce karta hai
$d_{model}^{- 0.5}$ se scaling: model size ke liye normalize karta hai

Why This Architecture Dominates

Parallelization: Sabhi positions simultaneously process hote hain (sequential RNNs ke mukable)
Constant path length: Koi bhi token pair directly attention ke through connected hai (RNN mein $O (n)$ steps ke mukable)
Learnable relationships: Attention weights data ke according adapt hote hain (fixed CNN kernels ke mukable)
Scalability: Compute badhne par aur layers/heads/dimensions stack karo

Recall Ek 12-Saal Ke Bachche Ko Explain Karo

Socho tum ek lambi kahani padh rahe ho. Tumhara brain ek word padhta hai, use bhool jaata hai, phir agla padhta hai — aisa nahi hota. Tumhari aankhein jump kar sakti hain — tum ek word padh sakte ho, phir shuru ki koi cheez yaad kar sakte ho, phir end se connect kar sakte ho. Transformers yehi karte hain!

Purane AI models (RNNs) aise the jaise ek tiny sliding window se padhna: ek word dekho, agla move karo, lekin pehla bhool jao. Transformers POORA sentence ek saath dekhte hain. Har word har doosre word ko dekhta hai aur decide karta hai "Is word ko samajhne ke liye mujhe us word par kitna dhyan dena chahiye?"

Do parts hain: Encoder (input sentence padhta hai, jaise French mein "Le chat noir") aur Decoder (output likhta hai, jaise "The black cat"). Encoder figure out karta hai ki sab kuch kya matlab rakhta hai. Decoder jawab banata hai, lekin ek special trick ke saath: word3 likhte waqt, ye sirf words 1 aur 2 dekh sakta hai (future words nahi dekhna!) — bilkul jaise tum apni diary likhte waqt kal nahi dekh sakte.

Jadui ingredient hai attention — ek formula jo "Word A aur Word B kitne related hain?" compute karta hai har pair ke liye. Ye lakhon baar training ke dauran karo, aur AI language patterns seekh leta hai!

Connections

Multi-Head Self-Attention — core mechanism jo encoder aur decoder dono mein use hota hai
Positional Encoding — Transformers sequence order kaise inject karte hain
Scaled Dot-Product Attention — fundamental attention formula
Encoder-Decoder Architecture — general seq2seq framework
Residual Connections — ResNet se gradient flow technique
Layer Normalization — stabilization technique
Teacher Forcing — autoregressive decoders ke liye training technique
Attention Masking — decoder mein causal masking
Tokenization — text ko input IDs mein convert karna
Beam Search — inference par decoding strategy

Flashcards

#flashcards/ai-ml

Transformer decoder ke har layer mein teen sub-layers kya hain?

(1) Masked multi-head self-attention, (2) Encoder-decoder cross-attention, (3) Position-wise feed-forward network. Har ek mein residual connection + LayerNorm hota hai.

Attention formula mein √d_k se scaling kyun use hoti hai?

Dot products dimension d_k ke saath magnitude mein badhte hain. Bade values softmax ko saturation regions mein push karte hain jahan gradients bahut chhote hote hain, learning hindered hoti hai. √d_k se divide karne par variance stable rehta hai.

Decoder mein causal masking ka purpose kya hai?

Position i ko training ke dauran future positions j > i attend karne se rokta hai. Ye autoregressive generation enforce karta hai: token i predict karne ke liye sirf tokens 1 se i-1 use ho sakte hain, future tokens nahi.

Transformer base model mein approximately kitne parameters hain?

~65 million parameters, input embedding, output embedding, aur pre-softmax projection ke beech weight tying use karke.

Original paper mein use ki gayi learning rate warmup schedule kya hai?

lr = d_model^(-0.5) × min(step^(-0.5), step × warmup_steps^(-1.5)) with warmup_steps=4000. 4000 steps tak linear increase, phir inverse square root decay.

Positional encoding add karne se pehle embeddings ko √d_model se multiply kyun karte hain?

Embeddings ki variance ~1 hoti hai, positional encodings bounded [-1,1] hoti hain. √d_model (~22.6 for d=512) se scale karna ensure karta hai ki embeddings initially dominate karein jabki PE positional information add kare bina semantic content ko overwhelm kiye.

Base model mein har attention head ki dimension kya hoti hai?

d_k = d_v = d_model / h = 512 / 8 = 64 dimensions per head.

Learned embeddings ki jagah sinusoidal positional encoding kyun use karte hain?

Sinusoids model ko training ke dauran dekhi gayi sequence lengths se lambi lengths tak extrapolate karne dete hain. PE(pos+k) ko PE(pos) ke linear function ke roop mein express kiya ja sakta hai, jo model ko relative positions seekhne mein help karta hai.

Base model mein FFN inner dimension d_ff kya hai?

2048 (model dimension 512 ka 4 guna). Upar project karta hai, ReLU apply karta hai, wapas neeche project karta hai.

Encoder-decoder attention, self-attention se kaise alag hai?

Cross-attention mein, Q decoder se aata hai, lekin K aur V encoder output se aate hain. Ye decoder ko target sequence generate karte waqt source sequence attend karne deta hai.

Label smoothing ε_ls=0.1 aur vocabulary size K ke saath, true class ko kya target probability milti hai?

1 − ε_ls + ε_ls/K. K=4 ke liye ye hai 0.9 + 0.025 = 0.925; baaki har class ko ε_ls/K = 0.025 milta hai.