Principal Component Analysis (PCA) theory — Hinglish notes

2.5.8 · AI-ML › Unsupervised Learning

PCA Actually Kya Karta Hai

Principal Component Analysis high-dimensional data ko ek nayi coordinate system mein transform karta hai jahan:

Pehla axis (1st principal component) maximum variance ki direction mein point karta hai
Doosra axis (2nd PC) maximum baachi hui variance ki direction mein point karta hai, pehle se orthogonal
Har agla axis poori variance ko maximize karta hai, pehle waale sab se orthogonal hokar

Kyun orthogonal? Hum chahte hain ki components variation ke independent sources capture karein. Orthogonality ensure karta hai koi redundancy nahi—har PC nayi information add karta hai.

Mathematics: PCA Scratch Se Derive Karna

Step 1: Data ko Center Karo (KYUN)

Data matrix $X \in R^{n \times d}$ ( $n$ samples, $d$ features) diya hai, pehle use center karo: $X_{c} = X - 1_{n} μ^{T}$ jahan $μ = \frac{1}{n} \sum_{i = 1}^{n} x_{i}$ mean vector hai.

Kyun center karein? PCA data ke "center of mass" se directions dhundta hai. Center kiye bina, pehla PC sirf mean ki taraf point kar sakta hai variance structure capture karne ki jagah. Centering coordinate origin ko data ka centroid bana deta hai.

Step 2: Variance Maximization Problem Formulate Karo

Hum ek unit vector $w_{1}$ (pehla PC) dhundhna chahte hain jis par data project karne se variance maximize ho.

$x_{i}$ ka $w_{1}$ par projection: $z_{i} = w_{1}^{T} x_{i}$

Projections ka variance: $Var (z) = \frac{1}{n} \sum_{i = 1}^{n} (z_{i} - \overset{z}{ˉ})^{2}$

Kyunki data centered hai, $\overset{z}{ˉ} = 0$ : $Var (z) = \frac{1}{n} \sum_{i = 1}^{n} (w_{1}^{T} x_{i})^{2} = \frac{1}{n} \sum_{i = 1}^{n} w_{1}^{T} x_{i} x_{i}^{T} w_{1}$

Yeh step kyun? Hum squared scalar projection expand kar rahe hain. Property use kar rahe hain $(a^{T} b)^{2} = a^{T} b b^{T} a$ jab result scalar ho.

Rearrange karke: $Var (z) = w_{1}^{T} (\frac{1}{n} \sum_{i = 1}^{n} x_{i} x_{i}^{T}) w_{1} = w_{1}^{T} C w_{1}$

jahan $C = \frac{1}{n} X_{c}^{T} X_{c}$ covariance matrix hai.

Step 3: Lagrange Multipliers Se Solve Karo

Hum chahte hain: $max_{w_{1}} w_{1}^{T} C w_{1} subject to ∥ w_{1} ∥ = 1$

Constraint kyun? Iske bina, $w_{1}$ ko scale karke variance arbitrarily bada kar sakte hain. Unit constraint ka matlab hai hum ek direction dhundh rahe hain, magnitude nahi.

Lagrangian: $L = w_{1}^{T} C w_{1} - λ (w_{1}^{T} w_{1} - 1)$

$w_{1}$ ke saath derivative leke zero par set karo: $\frac{\partial L}{\partial w _{1}} = 2 C w_{1} - 2 λ w_{1} = 0$

Derivative kyun? Hum critical points dhundh rahe hain. 2 ka factor quadratic forms differentiate karne se aata hai: $\frac{\partial}{\partial w} (w^{T} Aw) = 2 Aw$ jab $A$ symmetric ho.

Isse milta hai: $C w_{1} = λ w_{1}$

Yeh eigen-equation hai! $w_{1}$ ko $C$ ka eigenvector hona chahiye eigenvalue $λ$ ke saath.

Kaun sa eigenvector? Apne objective mein wapas substitute karo: $w_{1}^{T} C w_{1} = w_{1}^{T} (λ w_{1}) = λ w_{1}^{T} w_{1} = λ$

Captured variance $λ$ ke barabar hai! Variance maximize karne ke liye, sabse bade eigenvalue wala eigenvector choose karo.

Baad Wale Components Orthogonal Kyun Hote Hain

Doosre PC $w_{2}$ ke liye, hum variance maximize karte hain subject to:

$∥ w_{2} ∥ = 1$ (unit length)
$w_{2}^{T} w_{1} = 0$ (pehle PC se orthogonal)

Mathematical reason: Ek symmetric matrix (jaise covariance matrices) ke eigenvectors jo distinct eigenvalues se correspond karte hain, automatically orthogonal hote hain. Kyunki covariance matrices symmetric real matrices hain, inke paas orthonormal eigenvectors ka ek complete set hota hai.

Intuitive reason: Agar $w_{2}$ mein $w_{1}$ ke saath koi component hota, woh component woh variance capture karta jo pehle se $w_{1}$ ne capture kar liya hai (redundant). Orthogonality ensure karta hai ki har PC nayi, uncorrelated variance capture kare.

Reconstruction aur Dimensionality Reduction

Forward Transform (Encoding)

Data ko pehle $k$ PCs par project karo: $Z = X_{c} W_{k} \in R^{n \times k}$

Har row $z_{i}$ sample $x_{i}$ ki k-dimensional representation hai.

Reconstruction (Decoding)

Reduced representation se original data approximate karo: $\hat{X}_{c} = Z W_{k}^{T} = X_{c} W_{k} W_{k}^{T}$

Phir mean wapas add karo: $\hat{X} = \hat{X}_{c} + 1_{n} μ^{T}$

Reconstruction error (mean squared error): $MSE = \frac{1}{n} ∥ X_{c} - \hat{X}_{c} ∥_{F}^{2} = \sum_{i = k + 1}^{d} λ_{i}$

Yeh formula kyun? Frobenius norm squared un eigenvalues ka sum hai jo humne include nahi kiye. Har discarded PC apna eigenvalue error mein contribute karta hai.

Worked example Worked Example: 2D se 1D Reduction

Data: 2D mein 4 centered points $X_{c} = 2 - 1 - 2 1 1 - 0.5 - 1 0.5$

Step 1: Covariance matrix $C = \frac{1}{4} X_{c}^{T} X_{c} = \frac{1}{4} [105 5 2.5] = [2.5 1.25 1.25 0.625]$

Yeh step kyun? $X_{c}^{T} X_{c}$ compute karne se scatter matrix milta hai; $n$ se divide karne par covariance milta hai.

Step 2: Eigenvalues $det (C - λ I) = (2.5 - λ) (0.625 - λ) - 1.2 5^{2} = 0$ $λ^{2} - 3.125 λ = 0 ⟹ λ_{1} = 3.125, λ_{2} = 0$

Yeh step kyun? Characteristic equation un values dhundti hai jahan $C - λ I$ singular hota hai (koi inverse nahi hota).

Step 3: $λ_{1} = 3.125$ ke liye Eigenvector $(C - 3.125 I) w_{1} = 0$ $[- 0.625 1.25 1.25 - 2.5] [w_{11} w_{12}] = 0$

Pehli row se: $- 0.625 w_{11} + 1.25 w_{12} = 0 ⟹ w_{11} = 2 w_{12}$

Normalize karke: $w_{1} = \frac{1}{5} [21] \approx [0.894 0.447]$

Yeh step kyun? Homogeneous system solve karne se null space (eigenvector) milta hai. Normalization unit length ensure karta hai.

Step 4: Project karo $Z = X_{c} w_{1} = 2.236 - 1.118 - 2.236 1.118$

Interpretation: Har point ki principal direction ke along position. Pehla point PC ke along 2.236 units par hai.

Captured variance: $\frac{3.125}{3.125 + 0} = 100%$ (saari variance ek direction mein hai—data perfectly colinear hai!)

Mathematical Properties

1. PCA Reconstruction Error Minimize Karta Hai

$k$ components wala PCA Frobenius norm ke under $X_{c}$ ka best rank-k approximation deta hai: $W_{k} = ar g min_{W : W^{T} W = I_{k}} ∥ X_{c} - X_{c} W W^{T} ∥_{F}^{2}$

Kyun? Yeh Eckart-Young theorem hai. Best low-rank approximation SVD/eigendecomposition se aati hai.

2. Singular Value Decomposition Se PCA

Alternative computation: centered data ka SVD: $X_{c} = UΣ V^{T}$

Tab:

Principal components: $W = V$ (right singular vectors)
Eigenvalues: $λ_{i} = \frac{σ _{i}^{2}}{n}$ (squared singular values divided by $n$ )
Projected data: $Z = UΣ = X_{c} V$

SVD kyun? $X_{c}^{T} X_{c}$ explicitly compute karne se zyada numerically stable hai, especially jab $d ≫ n$ ho.

3. PC Space Mein Covariance

Projection ke baad, $Z$ ka covariance diagonal hota hai: $Cov (Z) = \frac{1}{n} Z^{T} Z = Λ_{k} = diag (λ_{1}, \dots, λ_{k})$

Matlab: Principal components uncorrelated hain. Humne data ko decorrelate kar diya hai.

Common mistake Common Mistake: "PCA Sabse Important Features Dhundta Hai"

Galat idea: "Agar mere paas 100 features hain aur main pehle 3 PCs rakhta hoon, toh PCA mujhe 3 sabse important original features batata hai."

Yeh sahi kyun lagta hai: Tum 100 se 3 par reduce kar rahe ho, toh lagta hai PCA features select kar raha hai.

Sach yeh hai: PCA nayi features create karta hai (sab originals ke linear combinations). Pehla PC sab 100 features ka weighted sum hota hai: $PC_{1} = w_{11} x_{1} + w_{12} x_{2} + \dots + w_{1, 100} x_{100}$ PCA feature extraction hai, feature selection nahi. Har PC mein har original feature involved ho sakta hai.

Yeh kab matter karta hai: Agar tumhe interpretability chahiye (jaise "kaun sa gene yeh disease cause karta hai?"), toh PCA components interpret karna mushkil hai. Iske bajaaye feature selection methods use karo (Lasso, mutual information, etc.).

Common mistake Common Mistake: Data Center Karna Bhool Jana

Galat approach:

pca = PCA(n_components=2)
Z = pca.fit_transform(X)  # X centered nahi hai!

Yeh sahi kyun lagta hai: Scikit-learn ka PCA automatically center karta hai, isliye log bhool jaate hain ki yeh zaroori hai.

Problem: Center kiye bina, pehla PC aksar origin se data cloud ke mean ki taraf point karta hai—ek component location par waste karta hai variance structure ki jagah.

Mathematical consequence: Covariance matrix $E [(x - μ) (x - μ)^{T}]$ ki jagah $E [x x^{T}]$ hoga. Pehle wale mein mean offset include hoti hai.

Fix: PCA se pehle hamesha center karo (ya verify karo ki tumhari library karta hai):

X_centered = X - X.mean(axis=0)

Common mistake Common Mistake: Different Features Ko Alag Tarah Scale Karne Ke Baad PCA Apply Karna

Scenario: Tumhare paas alag units mein features hain (age years mein, income dollars mein, height cm mein). Tum directly PCA apply karte ho.

Problem: PCA scale-invariant nahi hai. Bade numeric ranges wale features variance mein dominate karte hain:

Income: variance ≈ 10^9 (dollars squared)
Age: variance ≈ 100 (years squared) Pehla PC zyaadatar income variation capture karega, age ko ignore karke.

Yeh kyun matter karta hai: "Most variance" direction arbitrary unit choices par depend karta hai. Age days mein vs. years mein result change kar deta hai!

Fix: PCA se pehle features ko standardize (z-score) karo jab unke alag scales hon: $x_{ij}^{'} = \frac{x _{ij} - μ _{j}}{σ _{j}}$

Alternative view: Standardizing ka matlab hai PCA covariance matrix ki jagah correlation matrix par perform karna.

Recall 12-Saal-Ke Bachche Ko Samjhao

Socho tumhari class ek photo leti hai. Tumhare paas 3D space mein sabki position hai (left-right, forward-back, up-down). Lekin jab tum photo dekhte ho, tumhe ehsaas hota hai ki zyaadatar information sirf 2 dimensions mein hai—photo plane mein! Teesra dimension (depth) barely matter karta hai kyunki sab roughly ek hi distance par camera se khade the.

PCA data ke liye "photo angle" dhundne jaisa hai. Yeh tumhare data cloud ko dekhta hai aur poochta hai: "Agar mujhe photo leni hoti (kam dimensions par project karna hota), kaun sa angle sabse zyada information rakhega?" Pehla principal component camera ki main direction jaisa hai—yeh capture karta hai jahan log sabse zyada spread hain. Doosra PC side se dekhne jaisa hai koi remaining spread pakadne ke liye.

Hum kam dimensions kyun chahte hain? Socho sabki position describe karna. "3.2m left, 1.8m forward, 0.1m Sarah se upar" kehne ki jagah tum bas "Row 3, Position 7" keh sakte ho agar sab seedhi rows mein khade hain. Teen ki jagah do numbers! PCA messy data mein wahi "seedhi rows" dhundta hai.

Connections

Covariance Matrix — PCA isse decompose karke variance structure dhundta hai
Eigendecomposition — PCA ke neeche ka mathematical tool
Singular Value Decomposition — PCA compute karne ka alternative (zyada stable) tarika
Linear Autoencoders — Neural networks jo linear hone par PCA-jaisi representations seekhte hain
Kernel PCA — Kernel trick use karke nonlinear extension
Factor Analysis — PCA se related probabilistic model, explicit noise ke saath
Independent Component Analysis — Statistically independent (sirf uncorrelated nahi) components dhundta hai
Feature Scaling — Mixed units wale PCA ke liye zaroori preprocessing
Dimensionality Reduction — PCA ek technique hai; t-SNE, UMAP se compare karo
Variance — Woh quantity jo PCA successive orthogonal directions mein maximize karta hai

#flashcards/ai-ml

PCA ka goal kya hai? :: Data mein orthogonal directions dhundna jo variance maximize karein, jisse dimensionality reduction kiya ja sake aur zyaadatar information preserve ho.

PCA se pehle data center kyun karna chahiye?

Taaki pehla PC variance structure capture kare na ki sirf mean ki taraf point kare; PCA centroid ke around variance dhundta hai, origin se nahi.

Mathematically principal component kya hota hai?

Covariance matrix ka eigenvector; k-th PC woh eigenvector hai jo k-th sabse bade eigenvalue se correspond karta hai.

PCA mein eigenvalue kya represent karta hai?

Corresponding principal component dwara captured variance; bada eigenvalue = zyada variance explained.

PCA ke liye covariance matrix kaise compute karte hain?

C = \frac{1}{n} X_{c}^{T} X_{c}

jahan

X_{c}

centered data matrix hai (rows samples hain, columns features hain).

Principal components orthogonal kyun hote hain?

(1) Distinct eigenvalues wale symmetric matrices ke eigenvectors orthogonal hote hain. (2) Ensure karta hai ki har PC independent variance capture kare, koi redundancy nahi.

Pehle k principal components par data project kaise karte hain?

Z = X_{c} W_{k}

jahan

W_{k}

mein pehle k eigenvectors columns ke roop mein hain.

PCA mein reconstruction formula kya hai?

\hat{X}_{c} = Z W_{k}^{T} = X_{c} W_{k} W_{k}^{T}

, phir mean wapas add karo:

\hat{X} = \hat{X}_{c} + 1_{n} μ^{T}

k components rakhne par reconstruction error kya hota hai?

MSE = \sum_{i = k + 1}^{d} λ_{i}

— discarded eigenvalues ka sum squared error ke barabar hota hai.

k components kitni variance explain karte hain?

\frac{\sum _{i = 1}^{k} λ _{i}}{\sum _{i = 1}^{d} λ _{i}}

— rakhe gaye eigenvalues ke sum ka total sum se ratio.

Kya PCA feature selection hai ya feature extraction?

Feature extraction; yeh nayi features create karta hai (sab originals ke linear combinations), koi subset select nahi karta.

PCA se pehle features standardize kyun karte hain?

Kyunki PCA scale-invariant nahi hai; bade numeric scales wale features variance mein dominate karte hain. Standardizing sab features ko barabar footing par laata hai.

Covariance-based aur correlation-based PCA mein kya fark hai?

Covariance PCA raw data use karta hai; correlation PCA standardized data use karta hai (correlation matrix ke eigenvectors compute karne ke equivalent). Correlation use karo jab features alag units mein hon.

SVD ka PCA se kya relation hai?

X_{c} = UΣV^{T}

; principal components right singular vectors

V

hain, eigenvalues

λ_{i} = σ_{i}^{2} / n

hain. Zyada numerically stable hai.

PC space mein covariance structure kya hoti hai?

Diagonal —

Cov (Z) = Λ_{k}

. Principal components construction se uncorrelated hote hain.