Dimensionality Reduction Latent Variables

The unifying principle

Take data as a matrix $X \in \mathbb{R}^{n \times p}$ ($n$ samples, $p$ features), mean-centered. The empirical covariance is

$$C = \frac{1}{n} X^\top X \in \mathbb{R}^{p \times p}.$$

Diagonalize it:

$$C v_k = \lambda_k v_k, \qquad \lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_p \ge 0.$$

The eigenvectors $v_k$ are the principal directions; the eigenvalues $\lambda_k$ are the variance captured along each. If the spectrum drops sharply after $r$ terms ($\lambda_{r+1}\approx 0$), then

$$X \approx \sum_{k=1}^{r} \sigma_k\, u_k v_k^\top \quad (\text{truncated SVD}),\qquad r \ll p.$$

The $r$ retained coordinates are the latent variables (scores) $z = V_r^\top x$. This same eigen/SVD skeleton is what recurs everywhere — the only thing that changes is what the matrix $C$ physically means.

How it shows up in each field

AI-ML — Principal Component Analysis & latent embeddings

The form: exactly the derivation above. PCA solves

$$\max_{\|w\|=1} \operatorname{Var}(Xw) = w^\top C w \;\Rightarrow\; w = v_1.$$

Notation: scores $z = V_r^\top x$, reconstruction $\hat{x} = V_r z$. Nonlinear generalizations (autoencoders, VAEs) keep the idea but replace linear projection with a learned encoder $q(z\mid x)$; the bottleneck dimension $r$ is still the number of latent factors. Example: a $784$-pixel MNIST digit compresses to $\sim 50$ PCA components retaining $95\%$ variance; a VAE encodes it in a $\sim 10$-dim latent that smoothly interpolates between digits.

Maths — Spectral decomposition & low-rank approximation

The form: the underlying theorem is the Eckart–Young–Mirsky result — the best rank-$r$ approximation of $X$ in Frobenius (or spectral) norm is the truncated SVD:

$$\min_{\operatorname{rank}(B)\le r} \|X - B\|_F = \Big(\sum_{k>r}\sigma_k^2\Big)^{1/2}.$$

Notation: singular values $\sigma_k = \sqrt{n\lambda_k}$. This is the pure mathematical statement that "keep the top eigen-directions" is provably optimal, not just heuristic. It also connects to Mercer/Karhunen–Loève expansions of random functions. Example: a rank-1000 image matrix reconstructed from its top $30$ singular triplets looks nearly identical — the tail $\sigma_k$ carry little energy.

Physics — Order parameters & collective modes

The form: near a phase transition or in a many-body system, $10^{23}$ microscopic degrees of freedom collapse onto a few slow order parameters $\eta$ (magnetization $m$, condensate density, etc.). In Landau theory the free energy depends only on them:

$$F(\eta) = a(T)\,\eta^2 + b\,\eta^4 + \dots$$

The covariance-eigenvalue picture appears directly in normal modes (diagonalizing the Hessian of the potential) and in the spectral gap: dominant eigenvalues = slow, macroscopic modes; the rest are fast microscopic fluctuations integrated out. This is the renormalization-group intuition — irrelevant directions are the small-eigenvalue ones. Example: coupled oscillators $M\ddot{x} = -Kx$ diagonalize to independent normal modes; only the lowest-frequency (softest) modes govern long-time collective behavior.

Stock-Market — Risk factor models

The form: returns of $p$ assets are modeled as driven by $r$ common factors:

$$r_i = \alpha_i + \sum_{k=1}^{r}\beta_{ik} f_k + \varepsilon_i,\qquad r\ll p.$$

This is a latent-factor model; the $f_k$ are found either by economic construction (Fama–French: market, size, value) or by PCA on the return covariance matrix — the top eigenvector is essentially the market factor, and its eigenvalue explains most co-movement. The rest of the spectrum, near the Marchenko–Pastur random-matrix bulk, is treated as noise. Example: PCA on S&P 500 daily returns yields a dominant $v_1$ with all-positive loadings ("everything moves together") capturing $\sim 40\%$ of variance — the systematic risk factor.

Why this bridge matters

• Optimality transfers: Eckart–Young tells the ML engineer and the quant that truncating small eigenvalues is the mathematically best compression — so throwing away low-variance directions is principled, not arbitrary.
• Noise/signal separation transfers: Random-matrix theory (Marchenko–Pastur) developed in physics tells the quant which eigenvalues are real factors vs. sampling noise, and tells the ML practitioner when PCA components are meaningless.
• The "slow mode" intuition transfers: A physicist's instinct that dynamics live on a low-dimensional slow manifold is exactly the ML idea of a latent bottleneck and the finance idea of a few systematic factors driving co-movement.
• Interpretability transfers: Physics teaches that latent directions can be physical (order parameters); this pushes ML toward disentangled, meaningful latents and finance toward economically-named factors rather than opaque components.
• Danger transfers both ways: all these methods assume the important structure is linear/variance-dominant. Fat-tailed markets, nonlinear manifolds, and critical fluctuations break that assumption identically everywhere.

Connections

• 01-AI-ML/Principal-Component-Analysis
• 01-AI-ML/Autoencoders-and-VAEs
• 02-Maths/Singular-Value-Decomposition
• 02-Maths/Eckart-Young-Low-Rank-Approximation
• 03-Physics/Order-Parameters-and-Landau-Theory
• 03-Physics/Normal-Modes-and-Spectral-Gap
• 03-Physics/Random-Matrix-Theory-Marchenko-Pastur
• 04-Stock-Market/Factor-Models-Fama-French
• 04-Stock-Market/Covariance-and-Systematic-Risk

#bridge