Explained variance and choosing components

2.5.10 · AI-ML › Unsupervised Learning

Core Question

Jab hum PCA run karte hain aur principal components milte hain, hume kitne rakhne chahiye? Agar bahut kam rakhe toh information lose hoti hai; agar bahut zyada rakhe toh dimensionality reduction ka purpose hi khatam ho jaata hai. Explained variance hume iska quantitative jawab dene ka tarika deta hai.

[!intuition] The Big Picture

Socho tum ek 100-dimensional dataset compress kar rahe ho. Har principal component original data ki kuch "spread" (variance) capture karta hai. Pehla component sabse zyada variance capture karta hai, doosra uske baad ka sabse zyada (jo bachta hai usme se), aur aage bhi aise hi.

Key insight ye hai: Hume wo components nahi chahiye jo almost zero variance explain karte hain—wo mostly noise hote hain. Hum sirf itne components rakhna chahte hain jisse, say, 95% total variance capture ho jaye. Ye compression aur information retention ke beech balance banata hai.

Variance kyun important hai: Data space mein, variance = information. High variance wali directions woh hain jahan data points meaningfully alag hote hain. Low-variance directions aksar sirf noise ya irrelevant detail hoti hain.

[!definition] Explained Variance

$i$-ve principal component ka explained variance ratio hota hai:

${\text{EVR}}_i=\frac{{\lambda }_i}{{\sum }_{j=1}^d{\lambda }_j}$

jahan:

• ${\lambda }_i$ $i$-ve principal component ki eigenvalue (variance) hai
• $d$ original dimensions ki sankhya hai
• Denominator original data ki total variance hai

Iska matlab: ${\text{EVR}}_i$ hume batata hai ki component $i$ total variance ka kitna fraction capture karta hai.

Pehle $k$ components ka cumulative explained variance hota hai:

${\text{CEV}}_k={\sum }_{i=1}^k{\text{EVR}}_i=\frac{{\sum }_{i=1}^k{\lambda }_i}{{\sum }_{j=1}^d{\lambda }_j}$

Ye hume batata hai ki agar hum pehle $k$ components rakhte hain toh total variance ka kitna fraction capture hota hai.

[!formula] Derivation from First Principles

Step 1: Original data mein variance kya hai?

Original data matrix $\mathbf{X}$ (centered, $n\times d$) ki total variance har original dimension ke variance ka sum hoti hai:

$\text{Total Var}={\sum }_{j=1}^d\text{Var}(X_j)={\sum }_{j=1}^d\frac{1}{n}{\sum }_{i=1}^n(x_{ij}-{\bar{x}}_j{)}^2$

Agar $\mathbf{X}$ already centered hai (${\bar{x}}_j=0$), toh ye simplify hokar banta hai:

$\text{Total Var}=\frac{1}{n}\Vert \mathbf{X}{\Vert }_F^2$

jahan $\Vert \cdot {\Vert }_F$ Frobenius norm hai (saare squared entries ka sum).

Ye step kyun? Hume ek baseline chahiye: original data mein total "spread" kya hai jo hum preserve karne ki koshish kar rahe hain?

Step 2: Ek principal component se captured variance

Jab hum data ko principal component ${\mathbf{v}}_i$ (covariance matrix ka eigenvector) pe project karte hain, toh projected data ki variance hoti hai:

$\text{Var}(\mathbf{X}{\mathbf{v}}_i)=\frac{1}{n}(\mathbf{X}{\mathbf{v}}_i{)}^T(\mathbf{X}{\mathbf{v}}_i)=\frac{1}{n}{\mathbf{v}}_i^T{\mathbf{X}}^T\mathbf{X}{\mathbf{v}}_i$

Kyunki covariance matrix $\mathbf{C}=\frac{1}{n}{\mathbf{X}}^T\mathbf{X}$ hai aur ${\mathbf{v}}_i$ ek eigenvector hai eigenvalue ${\lambda }_i$ ke saath:

$\mathbf{C}{\mathbf{v}}_i={\lambda }_i{\mathbf{v}}_i$

Hume milta hai:

$\text{Var}(\mathbf{X}{\mathbf{v}}_i)=\frac{1}{n}{\mathbf{v}}_i^T(n\mathbf{C}){\mathbf{v}}_i={\mathbf{v}}_i^T\mathbf{C}{\mathbf{v}}_i={\mathbf{v}}_i^T({\lambda }_i{\mathbf{v}}_i)={\lambda }_i{\mathbf{v}}_i^T{\mathbf{v}}_i={\lambda }_i$

(Kyunki eigenvectors normalized hote hain: ${\mathbf{v}}_i^T{\mathbf{v}}_i=1$)

Ye step kyun? Eigenvalue ${\lambda }_i$ hi component $i$ ke along variance hai. Ye eigenvalues aur explained variance ke beech fundamental connection hai.

Step 3: Total variance eigenvalues ke sum ke barabar hai

Covariance matrices ki ek sundar property:

$\text{Total Var}=\text{trace}(\mathbf{C})={\sum }_{j=1}^d{\lambda }_j$

Kyun? Kisi matrix ka trace uske eigenvalues ke sum ke barabar hota hai (linear algebra ki ek identity). Covariance matrix ka trace original axes ke along variances ka bhi sum hota hai.

Ye step kyun? Ye hume hamare explained variance formula mein denominator deta hai. Data ki total variance saari eigenvalues ke sum ke barabar hai.

Step 4: Sab kuch ek saath

Component $i$ ke dwara explain ki gayi variance ka fraction:

${\text{EVR}}_i=\frac{\text{Variance of component }i}{\text{Total variance}}=\frac{{\lambda }_i}{{\sum }_{j=1}^d{\lambda }_j}$

Pehle $k$ components ke liye milake:

${\text{CEV}}_k=\frac{{\lambda }_1+{\lambda }_2+\cdots +{\lambda }_k}{{\sum }_{j=1}^d{\lambda }_j}$

Ye step kyun? Ye ratio hume information retention batata hai: agar ${\text{CEV}}_k=0.95$, toh hum $d$ se $k$ dimensions reduce karte hue 95% variance (information) rakh lete hain.

[!example] Worked Example 1: 3D se 2D Reduction

Setup: Hamare paas 3D data hai jiska covariance matrix hai:

$\mathbf{C}=\left[\begin{array}{ccc}4& 0& 0\\ 0& 1& 0\\ 0& 0& .1\end{array}\right]$

(Simplicity ke liye diagonal; generally, hum non-diagonal $\mathbf{C}$ ke eigenvectors compute karte)

Eigenvalues: ${\lambda }_1=4,{\lambda }_2=1,{\lambda }_3=0.1$

Step 1: Total variance ${\sum }_{j=1}^3{\lambda }_j=4+1+0.1=5.1$

Ye step kyun? Ye haara baseline hai—original 3D data ki 100% variance.

Step 2: Explained variance ratios ${\text{EVR}}_1=\frac{4}{5.1}\approx 0.784(78.4\%)$ ${\text{EVR}}_2=\frac{1}{5.1}\approx 0.196(19.6\%)$ ${\text{EVR}}_3=\frac{0.1}{5.1}\approx 0.020(2.0\%)$

Ye step kyun? Har ratio dikhata hai ki ek component kitni information carry karta hai.

Step 3: Cumulative explained variance ${\text{CEV}}_1=0.784$ ${\text{CEV}}_2=0.784+0.196=0.980(98\%)$ ${\text{CEV}}_3=0.980+0.020=1.000(100\%)$

Ye step kyun? Ye batata hai: agar hum 2 components rakhte hain, toh 98% variance retain hoti hai.

Decision: 2 components rakho. Hum sirf 2% information lose karte hain lekin dimensionality 33% kam ho jaati hai.

[!example] Worked Example 2: Image Compression

Setup: Ek 64×64 grayscale image jo 4096 dimensions mein flatten hui. PCA ke baad, pehle 10 eigenvalues hain:

$[1200,800,400,200,150,100,50,30,20,10]$

Saari eigenvalues ka sum 15,000 hai.

Step 1: Pehle component ka EVR ${\text{EVR}}_1=\frac{1200}{15000}=0.08(8\%)$

Ye step kyun? Sabse achha component bhi sirf 8% variance capture karta hai—images complex hoti hain!

Step 2: Pehle 5 components ka cumulative ${\text{CEV}}_5=\frac{1200+800+400+200+150}{15000}=\frac{2750}{15000}\approx 0.183(18.3\%)$

Ye step kyun? 5 components 20% se kam capture karte hain—reconstruction ke liye kaafi nahi.

Step 3: 90% variance ke liye kitne chahiye?

Hume ${\text{CEV}}_k\ge 0.90$ chahiye, toh ${\sum }_{i=1}^k{\lambda }_i\ge 0.90\times 15000=13500$.

Eigenvalues ko sum karte hue (practice mein, hamare paas sab hote aur hum cumulative sums compute karte), maan lo pehle 200 components se 13,500 milte hain.

Ye step kyun? Ye practical cutoff hai—hum 4096 se 200 dimensions compress kar sakte hain (95% reduction) aur 90% image detail rakh sakte hain.

[!example] Worked Example 3: Threshold se $k$ Choose Karna

Setup: Hum ek 50-dimensional dataset pe PCA run karte hain. Hume descending order mein eigenvalues milti hain. Hum 95% variance retain karne ke liye enough components rakhna chahte hain.

Algorithm:

cumsum = 0
total_var = sum(eigenvalues)
threshold = 0.95 * total_var
k = 0
 
for i, eigenvalue in enumerate(eigenvalues):
    cumsum += eigenvalue
    k = i + 1
    if cumsum >= threshold:
        break

Ye step kyun? Hum components ko importance ke order mein iterate karte hain, variance accumulate karte hain jab tak apna target nahi milta.

Example numbers: Agar eigenvalues hain [10, 8, 5, 3, 2, 1, 0.5, ...] (total = 50):

• 1 component ke baad: 10/50 = 20%
• 2 ke baad: 18/50 = 36%
• 3 ke baad: 23/50 = 46%
• 4 ke baad: 26/50 = 52%
• Tab tak continue karo jab tak cumulative ≥ 47.5 na ho (jo 50 ka 95% hai)

Ye approach kyun? Ye automatic aur data-driven hai. Koi guessing nahi, bas ek clear retention target.

[!mistake] Common Pitfalls

Mistake 1: "Pehla component 60% explain karta hai, toh mujhe sirf wahi rakhna chahiye"

Kyun sahi lagta hai: 60% ek majority lagta hai—aadhe se zyada!

Kya galat hai:

• High-dimensional data mein, achhi reconstruction ke liye aksar multiple components chahiye hote hain.
• 60% variance ≠ 60% "useful information." Baaki 40% mein downstream tasks (jaise classification) ke liye crucial structure ho sakta hai.
• Standard practice: sirf pehla component nahi, 90-95% cumulative variance ka target rakho.

Fix: Hamesha cumulative explained variance dekho aur ek threshold set karo (jaise 95%). Scree plot (eigenvalues vs. component number) use karo dekhne ke liye kahan returns kam hone lagte hain.

Steel-man: Ye galti variance ko ek simple percentage score ki tarah treat karne se aati hai. Reality mein, information components mein distributed hoti hai, aur data ki structure preserve karne ke liye hume unhe kaafi aggregate karna padta hai.

Mistake 2: "Component 10 ke baad eigenvalues chhoti hain, toh wo components useless hain"

Kyun sahi lagta hai: Chhoti eigenvalues matlab kam variance—noise lagta hai.

Kya galat hai:

• "Chhota" relative hota hai. Agar ${\lambda }_{11}=0.01$ lekin total variance 0.5 hai, toh ye total ka 2% hai—kuch toh hai!
• Kabhi kabhi lower components rare but important features capture karte hain (jaise outlier detection, fine-grained classes).

Fix: Absolute eigenvalue magnitudes ki jagah relative measures (explained variance ratio, cumulative %) use karo. Apna task bhi consider karo: visualization ke liye 2-3 components kaafi hain; reconstruction ke liye shayad bahut zyada chahiye.

Mistake 3: "Zyada components = better, toh main 90% original dimensions rakhunga"

Kyun sahi lagta hai: Zyada information better hai, aur data loss se bachate hain.

Kya galat hai:

• Tum PCA ka purpose hi khatam kar dete ho! Agar 90% dimensions rakhte ho, toh complexity almost kam hi nahi hui.
• Last components aksar signal nahi, noise capture karte hain. Inhe include karne se generalization kharab hoti hai (overfitting).

Fix: PCA compression aur noise reduction ke liye hai. Low-variance components aggressively drop karo. Ek achha rule: aisa $k$ rakho jisse ${\text{CEV}}_k\ge 0.95$ aur $k\ll d$ ho (jaise real compression ke liye $k\le 0.1d$).

[!recall]- Ek 12-Saal-Ke Bachche Ko Samjhao

Socho tumhare paas students ke baare mein 100 alag measurements ka ek bada dher hai: height, weight, test scores, hours studied, favorite color ek number ke roop mein, shoe size, aur 94 aur weird cheezein.

Ab, kuch measurements students ko samajhne ke liye really important hain (jaise test scores), aur kuch kaafi useless hain (jaise grades predict karne ke liye shoe size).

PCA ek smart organizer ki tarah hai jo naye "super-measurements" (principal components) banata hai. Pehla super-measurement purane measurements ko aise combine karta hai jo students ke beech SABSE ZYADA differences capture kare. Doosra uske baad ka sabse zyada capture karta hai, aur aage bhi aise hi.

Ye key hai: explained variance batata hai ki har super-measurement "students ke beech ke differences" ka kitna capture karta hai.

• Component 1 shayad 40% differences capture kare
• Component 2 shayad 25%
• Component 3 shayad 15%
• ...aur aage bhi

Ab, 100 measurements track karne ki jagah, tum pehle kuch super-measurements rakh sakte ho jo milake, say, 95% differences tak pahunch jaate hain. Shayad ye sirf 10 components hain! Tum student ke 100 numbers se 10 pe aa gaye, 95% information rakhte hue.

Choose kaise karte hain? Percentages add karte jao (cumulative explained variance) jab tak apna goal (jaise 95%) na mil jaaye. Itne hi components rakhte hain.

[!mnemonic] The Scree Plot Elbow

"Woh elbow dhundho jahan scree cliff se neeche girti hai"

Ek scree plot eigenvalues (y-axis) vs. component number (x-axis) dikhata hai. Ye ek pahad ki tarah lagta hai jisme pehle steep drop hoti hai aur phir ek lamba flat tail.

Mnemonic: Socho tum scree (dhile pathar) ek pahad ki dhlan se neeche daal rahe ho. Bade boulders (high eigenvalues) pehle girate hain aur pile up karte hain. Phir chhote pebbles (low eigenvalues) dheere neeche aate hain aur flat faile rehte hain.

Tum un components ko rakhna chahte ho jo plot ke flat hone se pehle hain—wahan "information boulders" hain. Flat tail sirf "noise pebbles" hai.

Elbow dhundho (woh bend jahan steep flat ho jaata hai). Elbow tak ke components rakho.

Choosing $k$ in Practice

Method 1: Cumulative Variance Threshold

• Ek target choose karo (jaise 95% ya 99%)
• Sabse chhota $k$ rakho jisse ${\text{CEV}}_k\ge$ target ho
• Best for: Reconstruction, compression tasks

Method 2: Scree Plot Elbow

• Eigenvalues vs. component index plot karo
• "Elbow" dhundho jahan curve flat ho jaata hai
• Elbow se pehle ke components rakho
• Best for: Exploratory analysis, visualization

Method 3: Cross-Validation (Supervised Tasks)

• Har $k$ ke liye, data ko $k$ dimensions mein reduce karo
• Reduced data pe model train karo
• Performance validate karo
• Woh $k$ choose karo jo validation accuracy maximize kare
• Best for: Classification, regression preprocessing

Method 4: Fixed Dimension Target

• Constraints ke basis pe $k$ decide karo (jaise "mujhe 2D plot chahiye" ya "main sirf 50 dimensions store kar sakta hoon")
• ${\text{CEV}}_k$ check karo dekhne ke liye kitni variance retain ho rahi hai
• Best for: Visualization (2D/3D), storage/speed constraints

Key Formulas Summary

| Concept | Formula | Interpretation | |---------|-------------| | Explained Variance Ratio | ${\text{EVR}}_i=\frac{{\lambda }_i}{{\sum }_{j=1}^d{\lambda }_j}$ | Component $i$ ke dwara captured variance ka fraction | | Cumulative Explained Variance | ${\text{CEV}}_k={\sum }_{i=1}^k{\text{EVR}}_i$ | Pehle $k$ components ke dwara captured total fraction | | Total Variance | ${\sum }_{j=1}^d{\lambda }_j=\text{trace}(\mathbf{C})$ | Saari eigenvalues ka sum = covariance ka trace | | Component Variance | $\text{Var}(\mathbf{X}{\mathbf{v}}_i)={\lambda }_i$ | Eigenvalue = us component ke along variance |

Connections

• Principal Component Analysis (PCA): Explained variance se hum decide karte hain ki PCA se kitne components rakhne hain
• Eigenvalues and Eigenvectors: Eigenvalues directly represent karte hain har principal component ke dwara captured variance
• Covariance Matrix: Covariance matrix ka trace data ki total variance ke barabar hota hai
• Dimensionality Reduction: Explained variance quantify karta hai ki jab hum dimensions reduce karte hain toh kitni information loss hoti hai
• Scree Plot: "Elbow" dhundh ke $k$ choose karne ka visual tool
• Singular Value Decomposition (SVD): Singular values eigenvalues se related hain; PCA via SVD ke liye ${\sigma }_i^2=n{\lambda }_i$
• Bias-Variance Tradeoff: Bahut zyada components rakhne se variance badhti hai (overfitting); bahut kam se bias badhta hai
• Feature Selection vs. Feature Extraction: Explained variance se decide hota hai ki kitne extracted features (components) use karne hain

#flashcards/ai-ml

Principal component $i$ ke liye explained variance ratio kya hai? :: ${\text{EVR}}_i=\frac{{\lambda }_i}{{\sum }_{j=1}^d{\lambda }_j}$, component $i$ ke dwara captured total variance ka fraction, jahan ${\lambda }_i$ uski eigenvalue hai.

Concept Map