Self-attention is the core operation that allows each position in a sequence to attend to all other positions, computing a weighted representation based on relevance. Unlike RNNs that process sequentially, self-attention enables parallel processing and long-range dependencies without the vanishing gradient problem.
Imagine you're reading a mystery story, and you come across the word "he." Who does "he" refer to? You have to look back through the previous sentences to find the answer—maybe it's the detective, or maybe it's the suspect.
Self-attention is how a computer does this! When the computer reads "he," it looks back at ALL the previous words and asks: "How much should I pay attention to each of these words to understand'he'?"
It gives each word a score (like 0% to 100%). If "detective" was mentioned recently and makes sense in context, it gets a high score like 80%. If "suspect" was mentioned but doesn't fit as well, it gets 20%. Then it combines the meanings of all these words based on their scores—80% of "detective's" meaning + 20% of "suspect's" meaning—to understand "he."
The cool part? It does this for EVERY word! Each word looks at every other word and decides how much to pay attention to it. That's why it's called self-attention—the sentence is paying attention to itself.
The computer learns HOW to score these words by practicing on millions of sentences until it gets really good at figuring out which words matter for understanding each word.
Positional Encoding - provides order information to self-attention
Attention Mechanism Basics - encoder-decoder attention (different from self-attention)
Computational Complexity - the O(n2) bottleneck comes from self-attention
Gradient Flow - self-attention enables better gradient flow than RNNs
#flashcards/ai-ml
What is self-attention and how does it differ from encoder-decoder attention? :: Self-attention is when a sequence attends to itself—each position can look at all other positions in the same sequence. Unlike encoder-decoder attention where queries come from decoder and keys/values from encoder, in self-attention Q, K, V all come from the same input sequence X through different learned projections.
Why do we need three separate matrices W^Q, W^K, W^V instead of using embedings directly?
The three projections create specialized representations: W^Q learns what aspects to search for, W^K learns what aspects to offer for matching, W^V learns what content to retrieve. This lets the model learn task-specific notions of similarity and relevance, rather than just using raw embedding similarity.
Derive why we scale attention scores by 1/sqrt(d_k)
Assume Q and K entries are random with mean 0, variance 1. The dot product q·k sums d_k terms, each with variance 1, so Var(q·k) = d_k. Standard deviation grows as sqrt(d_k). Without scaling, large d_k creates very large scores → softmax becomes nearly one-hot → gradients vanish. Dividing by sqrt(d_k) normalizes to unit variance, keeping softmax in the responsive region.
What is the computational complexity of self-attention and where does it come from?
O(n²d) where n is sequence length and d is model dimension. The n² comes from computing all pairwise attention scores (every position attends to every position), requiring an n×n attention matrix. This is the main bottleneck for long sequences.
Why can't self-attention learn word order without positional encodings?
Self-attention is permutation-invariant: the formula Attention(Q,K,V) = softmax(QK^T/sqrt(d_k))V treats positions as a set. If you shuffle inputs, outputs just shuffle correspondingly. Without positional encodings added to embeddings, "dog bites man" and "man bites dog" would produce identical representations (up to reordering).
In the formula softmax(QK^T/sqrt(d_k))V, explain what each matrix multiplication computes :: QK^T (n×d_k)(d_k×n) = (n×n) gives all-pairs similarity scores—each query against each key. softmax normalizes rows into probability distributions. The result (n×n) times V (n×d_v) = (n×d_v) performs weighted aggregation—each output position is a weighted sum of all value vectors, with weights from attention scores.
What happens to gradients in self-attention when d_k is large and we don't scale?
For large d_k, dot products become large (std grows as sqrt(d_k)). Large inputs to softmax create nearly one-hot outputs (e.g., [0.01, 0.98, 0.01]). When backpropagating through softmax, positions with near-zero probabilities have near-zero gradients → model can't learn to adjust attention to those positions → vanishing gradient problem.
How does self-attention solve the long-range dependency problem that RNNs struggle with?
RNNs process sequentially, so information from position1 must pass through all intermediate states to reach position 100(100 steps of potential degradation). Self-attention directly connects every position to every other in one step—position 100 can attend to position 1 with a direct path for gradients, avoiding the sequential bottleneck and vanishing gradients.
Chalo, self-attention ka core idea samajhte hain ek simple example se. Jab aap ek sentence padhte ho — "The animal didn't cross the street because it was too tired" — to "it" ka matlab samajhne ke liye aapko peeche "animal" pe dhyan dena padta hai, "street" pe nahi. Self-attention bilkul yahi kaam karta hai: har word ko allow karta hai ki woh sequence ke baaki sabhi words ko "dekh" sake aur decide kare ki kaunse words uske liye zyada important hain. Isme har word ek saath teen roles nibhata hai — Query (main kya dhoondh raha hoon), Key (main kya offer kar sakta hoon), aur Value (mera actual content kya hai). Yeh teen views hum input embeddings ko teen alag learned matrices (WQ,WK,WV) se multiply karke banate hain, taaki model khud seekh sake ki kya poochna hai aur kya return karna hai.
Ab mechanism ka math: Query aur Key ka dot product nikalte hain — agar do vectors similar direction mein point karte hain to unka dot product high hota hai, jo high relevance dikhata hai. Isse hamein ek n×n score matrix milta hai jahan har word ka har doosre word ke saath relevance score hota hai. Phir ek important trick — hum scores ko dk se divide karte hain. Reason yeh hai ki jaise-jaise dimension dk badhta hai, dot products ki magnitude bhi badhti jaati hai (variance dk ban jaata hai). Agar hum bade numbers pe seedha softmax lagayenge to distribution bahut zyada "peaked" ho jaayegi (almost one-hot), aur tab gradients bahut chote ho jaayenge — yaani vanishing gradient problem. Scaling se scores unit variance pe aa jaate hain aur training stable rehti hai.
Yeh matter kyun karta hai? Kyunki RNNs jaise purane models words ko ek-ek karke sequentially process karte the, jisse long-range dependencies pakadna mushkil tha aur training slow thi. Self-attention parallel processing allow karta hai — saare words ek saath process hote hain — aur direct connections banata hai kisi bhi do positions ke beech, chahe woh sentence mein kitne bhi door hon. Yahi reason hai ki Transformers itne powerful aur fast hain, aur aaj ke saare bade models (jaise GPT, BERT) isi core operation pe khade hain. Isliye agar aap modern AI samajhna chahte ho, to self-attention ka yeh intuition bilkul foundation hai.
