Feed-forward network sublayers — Hinglish notes

4.1.8 · AI-ML › Transformer Architecture

Feed-Forward Network (FFN) kya hai?

Architecture Deep Dive

Expansion-Contraction Pattern kyun?

Chaliye derive karte hain ki hum pehle expand kyun karte hain phir contract, shuru karte hain uss cheez se jo chahiye:

Goal: Ek non-linear function $f : R^{d_{model}} \to R^{d_{model}}$ seekhna jo complex patterns represent kar sake.

Step 1 — Universal Approximation

Universal approximation theorem ke mutabiq, ek two-layer network jisme kaafi hidden units hon, kisi bhi continuous function ko approximate kar sakta hai. Lekin kitne units chahiye?

Step 2 — Information Bottleneck

Agar $d_{f f} = d_{model}$ ho, toh ek bottleneck hai—limited capacity. Hidden layer ko complex combinations capture karne ke liye input se zyada features represent karne padte hain.

Step 3 — The 4x Rule

Empirically, $d_{f f} = 4 \times d_{model}$ achha kaam karta hai. Kyun?

Expansion: 4x zyada neurons ka matlab hai ki hum $\sim 4 \times d_{model}$ alag feature combinations compute kar sakte hain

Sparsity: ReLU activations sparse hote hain (~50% neurons active), toh effective capacity $\sim 2 \times d_{model}$ hoti hai

Compression: Second layer wapas project karta hai, network ko useful high-dimensional features seekhne par majboor karta hai

Activation Functions: ReLU → GELU → SiLU

Sirf ReLU kyun nahi?

Shuru karte hain ReLU (Rectified Linear Unit) se: $ReLU (x) = max (0, x)$

Problem: Zero par hard cutoff dead neurons create karta hai (gradient = 0 for $x < 0$ ).

GELU ki taraf evolution: $GELU (x) = x \cdot Φ (x) = x \cdot P (X \leq x), X \sim N (0, 1)$

Intuition: Input ko us probability se multiply karo ki woh ek random Gaussian se bada hai. Chhoti negative values ko chhota lekin non-zero gradient milta hai (smooth!).

Approximation: $GELU (x) \approx 0.5 x (1 + tanh [\frac{2}{π} (x + 0.044715 x^{3})])$

Modern SiLU/Swish: $SiLU (x) = x \cdot σ (x) = \frac{x}{1 + e ^{- x}}$

Better kyun? Har jagah smooth hai, self-gated hai (output input magnitude par depend karta hai), empirically large models ke liye zyada powerful.

Position-Wise = Cross-Position Mixing Nahi

Mathematical statement: Sequence $X = [x_{1}, x_{2}, \dots, x_{n}]$ ke liye jahan $x_{i} \in R^{d_{model}}$ : $FFN (X) = [FFN (x_{1}), FFN (x_{2}), \dots, FFN (x_{n})]$

Har $FFN (x_{i})$ same weights $W_{1}, W_{2}, b_{1}, b_{2}$ use karta hai lekin different input data ke saath.

Transformer Block Mein Role

Complete encoder block:

x^{'} x^{''} = LayerNorm (x + MultiHeadAttention (x)) = LayerNorm (x^{'} + FFN (x^{'}))

Yeh order kyun?

Pehle Attention: Context aggregate karo (linear mixing)
Phir FFN: Aggregated context se non-linear patterns extract karo
12-24 baar repeat karo: Har layer representations ko aur refine karta hai

Common Mistakes

Common mistake Mistake 1: "FFN sirf ek regular neural network layer hai"

Kyun sahi lagta hai: Yeh literally do linear layers hain activation ke saath—bilkul MLP jaisa dikhta hai!

Kya galat hai:

Regular MLP poore input ko ek saath process karta hai (jaise image flatten karo, vector ki tarah process karo)
FFN har position ko independently shared weights ke saath process karta hai
Yeh asliyat mein sequence modeling terms mein 1x1 convolution hai!

Fix: Socho "position-wise" = same network ko sequence dimension ke across broadcast karo. Code mein:

# Wrong mental model
output = mlp(x.view(-1))  # Flatten sequence
# Correct
output = torch.stack([mlp(x[i]) for i in range(seq_len)])
# Or efficiently: mlp(x)  # broadcasts automatically

Common mistake Mistake 2: "4x expansion arbitrary hai"

Kyun sahi lagta hai: Ek magic number lagta hai; 2x ya 8x kyun nahi? Kya galat hai: Iske peeche gehri reasoning hai:

Bahut chhota (jaise 1x): Autoencoder ban jaata hai—compress karo phir reconstruct karo, limited capacity
Bahut bada (jaise 8x): Zyada parameters → slower, overfitting risk, minimal accuracy gain
4x sweet spot: Hazaron experiments mein empirically test kiya gaya. Balance karta hai:
- Expressiveness (complex functions seekhne ke liye enough hidden units)
- Efficiency (parameters $O (d^{2})$ scale karte hain, toh 8x parameters double kar dega <5% gain ke liye)
- Gradient flow (bahut wide layers mein vanishing gradient issues ho sakte hain)

Fix: Ise ek empirical result ke roop mein samjho jo theory se backed hai (universal approximation + sparsity), random choice nahi. Modern models kabhi kabhi 3x ya 5.33x use karte hain (MoE models), lekin 4x default baseline hai.

Common mistake Mistake 3: "FFN output ka magnitude input ke similar hona chahiye"

Kyun sahi lagta hai: Residual connection FFN output ko input se add karta hai, toh woh comparable hone chahiye.

Kya galat hai:

Addition ke baad LayerNorm result ko normalize karta hai
FFN output magnitude naturally vary karta hai learned features ke basis par
Output magnitude ko over-constrain karna (jaise use zero ke paas force karna) learning rokta hai

Jo asliyat mein matter karta hai: Gradient achha flow karna chahiye, activation magnitude nahi. Residual connections ensure karte hain ki gradients dono branches (attention + FFN) mein flow karein, chahe activation scales kuch bhi hon.

Fix: Architecture par trust karo. LayerNorm + residuals scale handle karte hain. Gradient health par focus karo (GELU/SiLU use karo, proper initialization karo).

Modern Variations

GLU Variants (Gated Linear Units)

Recent models (PaLM, LLaMA) standard FFN ko gated variants se replace karte hain:

$FFN_{GLU} (x) = (σ (x W_{1}) ⊙ x W_{3}) W_{2}$

Jahan $⊙$ element-wise multiplication hai.

Gating kyun?

$σ (x W_{1})$ ek learned gate ki tarah kaam karta hai (kaun se features pass karne hain)
$x W_{3}$ features compute karta hai
Gate × Features = selective amplification
Benefit: Same performance ke liye 15-20% kam parameters (gates implicit regularization provide karte hain)

Mixture of Experts (MoE)

Single FFN ko multiple expert FFNs + router se replace karo:

$FFN_{MoE} (x) = \sum_{i = 1}^{N} Router (x)_{i} \cdot Expert_{i} (x)$

Kyun?

Har expert specialize karta hai (jaise Expert 1 = syntax, Expert 2 = semantics)
Sparse activation: sirf top-k experts har token ke liye compute karte hain
Benefit: 10x parameters, 1.5x compute (zyaadaatar experts har token ke liye dormant rehte hain)

Connections

Multi-Head Attention: Attention information mix karta hai, FFN use process karta hai
Residual Connections: FFN output input mein add hota hai, deep networks enable karta hai
Layer Normalization: Residual ke baad apply hota hai, FFN training stabilize karta hai
Positional Encoding: FFN position encode nahi karta—input ke positional encoding par rely karta hai
Transformer Encoder Block: FFN attention ke baad second sub-layer hai
Activation Functions: Choice (ReLU/GELU/SiLU) performance par significantly impact karti hai
Universal Approximation Theorem: Two-layer FFN structure ko justify karta hai
Batch Normalization vs Layer Normalization: Sequences ke liye BatchNorm ki jagah LayerNorm use hota hai

Flashcards

#flashcards/ai-ml

Transformers mein feed-forward network sublayer kya hai? :: Ek position-wise fully connected network jo har token par independently apply hota hai, do linear layers se bana hota hai jiske beech ek non-linear activation hota hai, typically dimension 4x expand karta hai phir wapas project karta hai.

FFN model dimension se 4x tak expand kyun karta hai?

Complex non-linear patterns seekhne ke liye ek richer feature space create karne ke liye. Expansion universal approximation ke liye enough capacity provide karta hai, jabki ReLU sparsity (~50% active) effective capacity ko ~2x model dimension banata hai.

Standard Transformer FFN ka formula kya hai?

FFN(x) = max(0, xW₁ + b₁)W₂ + b₂, jahan W₁ dimensions expand karta hai (d_model → d_ff), activation apply hota hai, aur W₂ wapas project karta hai (d_ff → d_model).

FFN position-wise kyun apply hota hai?

Attention already positions ke beech information mix kar chuka hota hai. FFN position-independent feature extraction mein specialize karta hai already-mixed representations par, jo perfectly parallelize hota hai aur redundant cross-position modeling rokta hai.

Transformer parameters ka kitna percentage FFN mein hota hai?

Har layer ke parameters ka approximately 2/3. d_ff = 4×d_model ke saath, FFN ke paas ~8d²_model parameters hote hain vs attention ke ~4d²_model.

Models ReLU se GELU se SiLU ki taraf kyun gaye?

ReLU mein hard cutoff hai (x<0 ke liye dead neurons). GELU activation ko Gaussian probability weighting se smooth karta hai. SiLU (x·σ(x)) self-gated, har jagah smooth hai, aur empirically large models ke liye best perform karta hai.

FFN aur regular MLP mein kya fark hai?

FFN same network ko har sequence position par independently apply karta hai (1x1 convolution ki tarah), jabki MLP poore input ko ek single vector ki tarah process karta hai. FFN mein position-wise weight sharing hota hai.

GLU variant kya hai aur ise kyun use karte hain?

Gated Linear Unit: FFN_GLU(x) = (σ(xW₁) ⊙ xW₃)W₂, jahan ek path doosre ko gate karta hai. Learned selective amplification ke through same performance ke liye 15-20% parameter savings provide karta hai.

Recall Ek 12-saal ke bacche ko samjhao

Socho tum class mein ho aur teacher sabse ek interesting fact share karne ko kehti hai. Yeh attention ki tarah hai—sabne suna kya logon ne kaha aur apne dimaag mein un ideas ko mila liya.

Ab, har student ghar jaata hai aur un saari baaton ke baare mein gehraai se sochta hai jo sunni thi, unhe apni pehle ki jaankari se jodata hai. Shayad tune suna "dogs loyal hote hain" aur "wolves pack mein shikaar karte hain," aur ghar par tune realize kiya "oh, dogs wolves se evolve hue, isliye woh social hote hain!" Woh gehri soch akele ghar par hi feed-forward network hai.

Tu sochte waqt apne classmates ko call nahi karta (doosron ke saath koi mixing nahi). Tu sirf woh process karta hai jo tune seekha. Phir kal, tu class mein wapas aata hai behtar samajh ke saath, aur cycle repeat hoti hai.

"4x expansion" waise hai jaise ghar par sochte waqt 4x zyada brain space milna—tu bahut zyada connections aur ideas consider kar sakta hai apna final thought class mein wapas laane se pehle.