t-SNE for visualization — Hinglish notes

2.5.11 · AI-ML › Unsupervised Learning

The Core Idea

Goal: High-dimensional points $x_{1}, \dots, x_{n} \in R^{d}$ ko low-dimensional points $y_{1}, \dots, y_{n} \in R^{2}$ (ya 3D) mein map karo taaki similar points similar rahein.

Kaise?

High-D space mein ek probability $p_{ij}$ define karo ki point $i$ , point $j$ ko apna neighbor chunega (Gaussian similarity ke basis par).
Low-D space mein ek probability $q_{ij}$ define karo ki $i$ , $j$ ko chunega (1 degree of freedom wali t-distribution use karte hain, jiske heavier tails hote hain).
KL-divergence minimize karke $q_{ij}$ ko $p_{ij}$ se match karo: $KL (P ∥ Q) = \sum_{i \neq = j} p_{ij} lo g \frac{p _{ij}}{q _{ij}}$ .

Low-D mein t-distribution kyun? Crowding problem: 100D mein aap bahut saare points ek center se equidistant rakh sakte ho. 2D mein woh overlap kar jaate. T-distribution ke heavy tails moderately-distant points ko zyada spread out hone dete hain, jisse overlap kam hoti hai.

Step-by-Step Derivation

Step 1: High-Dimensional Similarities $p_{ij}$

Har point $i$ ke liye, hum conditional probability calculate karte hain ki $i$ , $j$ ko neighbor chunega:

p_{j ∣ i} = \frac{exp ( - ∥ x _{i} - x _{j} ∥ ^{2} /2 σ _{i}^{2} )}{\sum _{k \neq = i} exp ( - ∥ x _{i} - x _{k} ∥ ^{2} /2 σ _{i}^{2} )}

Gaussian kyun? Hum similarity ko $x_{i}$ par centered Gaussian ke roop mein model karte hain. $σ_{i}$ (practice mein perplexity kehte hain) neighborhood size control karta hai.

Symmetrize karo taaki joint probability mile:

p_{ij} = \frac{p _{j ∣ i} + p _{i ∣ j}}{2 n}

Symmetrize kyun? Taaki $p_{ij} = p_{j i}$ ho (koi direction bias nahi), aur $\sum_{ij} p_{ij} = 1$ ho (valid probability distribution).

Step 2: Low-Dimensional Similarities $q_{ij}$

2D mein, Student's t-distribution (1 degree of freedom) use karo:

q_{ij} = \frac{( 1 + ∥ y _{i} - y _{j} ∥ ^{2} ) ^{- 1}}{\sum _{k \neq = l} ( 1 + ∥ y _{k} - y _{l} ∥ ^{2} ) ^{- 1}}

T-distribution kyun? Low-D mein Gaussian se compare karo:

Gaussian: $e^{- ∥ y_{i} - y_{j} ∥^{2}}$ → fast decay hota hai, door ke points vanish ho jaate hain.
t-distribution: $(1 + ∥ y_{i} - y_{j} ∥^{2})^{- 1}$ → slower decay hota hai (heavy tail), door ke points contribute karte rehte hain, unhe spread hone ki jagah milti hai.

Step 3: KL Divergence Minimize Karo

Cost function hai:

C = KL (P ∥ Q) = i \neq = j \sum p_{ij} lo g \frac{p _{ij}}{q _{ij}}

Is direction mein kyun (na ki $KL (Q ∥ P)$ )? $KL (P ∥ Q)$ heavily penalize karta hai $q_{ij} ≪ p_{ij}$ ko (nearby points ko push apart karna), lekin lightly penalize karta hai $q_{ij} ≫ p_{ij}$ ko (door ke points jo bahut paas hain). Hum local structure (nearby points) preserve karne ki zyada parwah karte hain, toh yeh asymmetry sahi hai.

Gradient:

\frac{\partial C}{\partial y _{i}} = 4 j \neq = i \sum (p_{ij} - q_{ij}) (y_{i} - y_{j}) (1 + ∥ y_{i} - y_{j} ∥^{2})^{- 1}

Yeh gradient kyun?

$(p_{ij} - q_{ij})$ : Agar $p_{ij} > q_{ij}$ , toh points $i, j$ 2D mein bahut door hain → paas kheencho. Agar $p_{ij} < q_{ij}$ , toh woh bahut paas hain → door dhakelo.
$(y_{i} - y_{j})$ : force ki direction.
$(1 + ∥ \dots ∥^{2})^{- 1}$ : t-distribution weight, smooth gradients ensure karta hai.

Hum gradient descent se optimize karte hain (aksar momentum + early exaggeration trick ke saath).

Hyperparameters

Doosre important parameters:

Learning rate (typically 100–1000): bahut low = slow convergence, bahut high = instability.
Number of iterations (1000–5000): t-SNE gradient descent se optimize karta hai, bahut saare steps chahiye.
Early exaggeration (pehle ~250 iterations): $p_{ij}$ ko 4 se multiply karo taaki tight, well-separated clusters banein, phir fine-tuning ke liye 1 par wapas aao.

Worked Example 1: 3D Gaussian Blobs to 2D

Data: 3D mein 3 clusters, har cluster ek Gaussian blob hai jisme 100 points hain.

Cluster A: centered at (0,0,0), σ=0.5
Cluster B: centered at (5,0,0), σ=0.5
Cluster C: centered at (2.5, 4, 0), σ=0.5

Step 1: Saare pairs ke liye $p_{ij}$ compute karo (300 × 299 / 2 = 44,850 pairs).

Cluster A ke do points ke liye, $∥ x_{i} - x_{j} ∥^{2} \approx 0.5$ , toh $p_{ij}$ bada hoga.
A ke ek point aur B ke ek point ke liye, $∥ x_{i} - x_{j} ∥^{2} \approx 25$ , toh $p_{ij} \approx 0$ .

Step 2: $y_{i}$ ko 2D mein randomly initialize karo (maan lo, $[- 0.001, 0.0001]$ mein uniform).

Step 3: $q_{ij}$ compute karo (initially sab similar, ~1/44850).

Step 4: Gradient compute karo. Example: agar cluster A ka point $i$ 2D mein apne true neighbor $j$ se door hai, toh $p_{ij} - q_{ij} > 0$ , gradient unhe paas kheenchta hai.

Yeh step kyun? Gradient descent iteratively $y_{i}$ adjust karta hai taaki 2D distribution $Q$ , high-D distribution $P$ se match kare.

Iterations: 1000 steps ke baad, clusters A, B, C 2D mein 3 distinct blobs form karte hain. Global distances (A-to-B vs A-to-C) distort ho sakti hain, lekin within-cluster aur between-cluster separation preserve hoti hai.

Worked Example 2: MNIST Digits

Data: 60,000 images, har ek 784 pixels ki (28×28 flattened).

Perplexity = 30: Har digit ~30 neighbors consider karta hai.

Process:

60k points ke liye $p_{ij}$ compute karo (sparse: sirf top neighbors per point store karo, total ~1.8 billion pairs → speed ke liye Barnes-Hut jaise approximate methods use karo).
5000 iterations ke liye gradient descent run karo.
Result: 2D mein 10 distinct clusters (har digit 0-9 ke liye ek). Kuch overlap (jaise, 4 aur 9 visually similar hain).

Yeh useful kyun hai? Ab ek insaan visually inspect kar sakta hai learned representation ko: Kya koi outliers hain? Kya digits coherent clusters form karte hain? Kya similar digits ke beech smooth transition (manifold) hai?

Common Mistakes

Comparison: PCA vs t-SNE

| Aspect | PCA | t-SNE | |--------|----| | Preserves | Global variance (linear) | Local neighborhoods (non-linear) | | Deterministic? | Haan | Nahi (random init) | | Speed | Fast ( $O (n d^{2})$ ) | Slow ( $O (n^{2})$ , ~ $O (n lo g n)$ Barnes-Hut ke saath) | | Interpretable axes? | Haan (principal components) | Nahi (arbitrary coordinates) | | Use case | Quick EDA, preprocessing | Publication/EDA ke liye final visualization |

Algorithm Pseudocode

1. Compute pairwise squared distances in high-D: D[i,j] = ||x_i - x_j||²
2. For each point i:
     Binary search σ_i to achieve desired perplexity
     Compute p_j|i = exp(-D[i,j] / 2σ_i²) / Σ_k exp(-D[i,k] / 2σ_i²)
3. Symmetrize: p_ij = (p_j|i + p_i|j) / 2n
4. Initialize y_i ~ N(0,10⁻⁴ I) randomly in 2D
5. For t = 1 to max_iter:
     Compute q_ij = (1 + ||y_i - y_j||²)⁻¹ / Z
     Compute gradient: ∂C/∂y_i = 4 Σ_j (p_ij - q_ij)(y_i - y_j)(1 + ||y_i - y_j||²)⁻¹
     Update y_i via gradient descent with momentum
     (Optional: early exaggeration in first 250 iters)
6. Return y_1, .., y_n (2D coordinates)

t-SNE Kyun Kaam Karta Hai: Math Intuition

Crowding problem: 100D mein, radius $r$ wale sphere ka volume $\propto r^{100}$ hota hai. 2D mein, volume $\propto r^{2}$ hota hai. Toh 100D neighbors saare 2D mein equidistant nahi fit ho sakte → woh crowd/overlap kar jaate hain.

Solution: T-distribution $(1 + d^{2})^{- 1}$ bade $d$ ke liye $d^{- 2}$ ki tarah decay karta hai, jo Gaussian ke $e^{- d^{2}}$ se bahut slower hai. Yahi heavy tails deta hai: 2D mein moderately-far points ka $q_{ij}$ phir bhi non-negligible ho sakta hai, toh woh kam repel karte hain aur naturally spread out ho jaate hain.

Force analogy: Har pair $(i, j)$ ek force exert karta hai:

Attractive agar $p_{ij} > q_{ij}$ (unhe aur paas hona chahiye).
Repulsive agar $p_{ij} < q_{ij}$ (unhe aur door hona chahiye).
Equilibrium $p_{ij} = q_{ij}$ KL ka local minimum hai.

Recall Feynman Explanation (ELI12)

Socho tumhare paas ek giant map hai ek desh ka jisme 100 roads har city ko connect karti hain. Tum ise paper par ek simple 2D map mein draw karna chahte ho. Problem: tum saari 100 roads ko bina overlap ke fit nahi kar sakte!

t-SNE ek smart artist ki tarah hai jo kehta hai: "Main close friends (neighboring cities) ko paper par paas rakhuga. Door ki cities? Unhe kahi door push kar dunga, chahe exact distance galat ho."

Yeh kaam kyun karta hai? Artist ek magic pen (t-distribution) use karta hai jo moderately-far cities ko smoothly spread out hone deta hai bina ek doosre mein crash kiye. Ek normal pen (Gaussian) unhe overlap kara deta.

Toh final map neighborhoods clearly dikhata hai (tum clusters dekh sakte ho), lekin clusters ke beech ki distance squishy hai. Ruler se mat mapo—bas view enjoy karo!

Connections

2.5.1 PCA: Linear dimensionality reduction, global variance preserve karta hai. t-SNE non-linear hai, local structure preserve karta hai.
2.5.10 UMAP: Similar goal (local structure preservation), lekin faster aur hyperparameters ke saath deterministic. Aksar t-SNE se compare kiya jaata hai.
2.4.3 K-Means Clustering: t-SNE visualization ke baad, tum original high-D data par formally cluster karne ke liye k-means use kar sakte ho.
3.2.1 Autoencoders: Neural networks ke zariye non-linear dimensionality reduction. Visualization + downstream tasks ke liye use ho sakta hai (t-SNE ke unlike).
1.3.5 KL Divergence: t-SNE mein core loss function hai. KL(P||Q) asymmetry samajhna crucial hai.
2.5.9 Manifold Learning: t-SNE assume karta hai ki data high-D space mein ek low-D manifold par hai.

When to Use t-SNE

✅ Use karo jab:

Tumhare paas high-dimensional data hai (50+ features) aur tum human interpretation ke liye clusters visualize karna chahte ho.
Modeling se pehle data explore karna ho (EDA).
Tum paper/report mein results present kar rahe ho aur ek compelling 2D scatter plot chahiye.

❌ Mat use karo jab:

Tumhe reproducible, deterministic embeddings chahiye (PCA ya UMAP use karo).
Tumhe naya test data embed karna ho (t-SNE ka koi inverse mapping nahi; parametric t-SNE ya autoencoders use karo).
Data already low-D ho (3–5 features): bas scatter plots ya PCA use karo.
Tumhe downstream ML tasks ke liye embeddings chahiye (classification, regression): PCA, LDA, ya autoencoders use karo.

Practical Tips

Perplexity: 30 se shuru karo. Bade datasets ke liye badhao (1M points → perplexity 50–100), chhote datasets ke liye ghataao (100 points → perplexity 5–15).
Multiple runs: t-SNE stochastic hai. 3–5 baar alag seeds ke saath run karo aur consistency check karo. Agar results bahut zyada vary karein, toh data mein clear structure nahi hogi.
Preprocessing: t-SNE se pehle features standardize karo (mean=0, std=1). High-variance features distances dominate karte hain.
Coloring: Known labels hain toh points ko unse color karo (agar available ho) taaki cluster separation validate ho sake.
Barnes-Hut approximation: $n > 5000$ ke liye, Barnes-Hut t-SNE use karo (scikit-learn mein available hai) taaki complexity $O (n^{2})$ se $O (n lo g n)$ ho jaaye.

#flashcards/ai-ml

t-SNE ka full form kya hai? :: t-Distributed Stochastic Neighbor Embedding

t-SNE ka primary goal kya hai?

High-dimensional data ko 2D/3D mein map karna jabki local neighborhood structure preserve ho (similar points paas rahein).

t-SNE low-D mein Gaussian ki jagah t-distribution kyun use karta hai?

T-distribution ke heavier tails hote hain, jo crowding problem solve karta hai aur moderately-far points ko naturally spread out hone deta hai bina overlap ke.

t-SNE kaun sa cost function minimize karta hai?

KL divergence:

KL (P ∥ Q) = \sum_{i \neq = j} p_{ij} lo g \frac{p _{ij}}{q _{ij}}

, jahan

P

high-D similarity hai aur

Q

low-D similarity hai.

t-SNE mein perplexity kya hai?

Ek hyperparameter jo control karta hai ki har point kitne effective neighbors consider karta hai (typically 5–50). Yeh

2^{H (P_{i})}

hai jahan

H

conditional distribution ki entropy hai.

t-SNE stochastic kyun hai?

Yeh low-D coordinates ki random initialization aur gradient descent use karta hai, isliye alag runs alag embeddings dete hain (lekin cluster structure consistent honi chahiye).

Kya tum t-SNE embeddings par classifier train kar sakte ho?

Nahi. t-SNE sirf visualization ke liye hai—yeh non-parametric hai (naye data ke liye koi explicit mapping nahi) aur stochastic hai. Downstream tasks ke liye PCA ya autoencoders use karo.

Gradient term

(p_{ij} - q_{ij})

hame kya batata hai?

Agar

p_{ij} > q_{ij}

, toh points

i, j

2D mein bahut door hain (attractive force unhe paas kheenchti hai). Agar

p_{ij} < q_{ij}

, toh woh bahut paas hain (repulsive force unhe door dhakeli hai).

t-SNE plot mein global distances interpret kyun nahi karne chahiye?

t-SNE sirf local structure preserve karta hai. Clusters ke beech ki distance arbitrary aur distorted ho sakti hai. Sirf cluster separation aur density meaningful hain.

Crowding problem kya hai?

High dimensions mein, bahut saare points ek center se equidistant ho sakte hain (large volume). 2D mein, woh overlap kar jaate (small volume). t-SNE ki t-distribution heavy tails se yeh solve karta hai.

t-SNE mein high-D conditional probability formula kya hai?

p_{j ∣ i} = \frac{e x p ( - ∥ x _{i} - x _{j} ∥ ^{2} /2 σ _{i}^{2} )}{\sum _{k \neq = i} e x p ( - ∥ x _{i} - x _{k} ∥ ^{2} /2 σ _{i}^{2} )}

t-SNE mein low-D similarity formula kya hai? :: $q_{ij} = \frac{( 1 + ∥ y _{i} - y _{j} ∥ ^{2} ) ^{- 1}}{\sum _{k \neq = l} ( 1 + ∥ y _{k} - y _{l} ∥ ^{2} ) ^{- 1}}$

t-SNE mein early exaggeration kya hai?

Pehle ~250 iterations mein

p_{ij}

ko 4 se multiply karna taaki tight, well-separated clusters banein, phir fine-tuning ke liye 1 par wapas aana.

t-SNE ki typical complexity kya hai?

Exact computation ke liye

O (n^{2})

. Barnes-Hut approximation bade datasets ke liye ise

O (n lo g n)

tak reduce karta hai.

t-SNE ko PCA ke upar kab use karna chahiye?

Jab complex, non-linear cluster structures visualize karne ho. PCA quick EDA, preprocessing, ya interpretable axes ke liye better hai.