Eigenvalues and eigenvectors

WHAT are we even asking?

• WHY nonzero? $v=0$ trivially satisfies $A\cdot 0 = \lambda \cdot 0$ for any $\lambda$, so it tells us nothing. We forbid it.
• WHY does $\lambda$ matter in ML? PCA, covariance structure, PageRank, stability of gradient descent, spectral clustering — all are "find the special directions" problems.

HOW do we find them? (Derivation from scratch)

We want $Av = \lambda v$ with $v \neq 0$. Rearrange:

$Av - \lambda v = 0 \;\Rightarrow\; Av - \lambda I v = 0 \;\Rightarrow\; (A - \lambda I)\,v = 0$

Now the key logical step. $(A-\lambda I)v = 0$ is a homogeneous system.

• If $(A-\lambda I)$ were invertible, the only solution would be $v = (A-\lambda I)^{-1}0 = 0$. But we banned $v=0$.
• So for a nonzero $v$ to exist, $(A-\lambda I)$ must be singular (non-invertible).
• A matrix is singular $\iff$ its determinant is zero.

Then for each $\lambda$, plug back into $(A-\lambda I)v = 0$ and solve for the null-space vectors $v$.

Worked Example 1 — a clean $2\times 2$

$A = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$

Step 1: build $A-\lambda I$. Why? We derived that eigenvalues make this matrix singular. $A - \lambda I = \begin{pmatrix} 2-\lambda & 1 \\ 1 & 2-\lambda \end{pmatrix}$

Step 2: determinant = 0. Why? Singular $\iff \det = 0$. $(2-\lambda)^2 - (1)(1) = 0$ $\lambda^2 - 4\lambda + 3 = 0 \;\Rightarrow\; (\lambda-1)(\lambda-3)=0$ So $\lambda_1 = 1,\ \lambda_2 = 3$.

Step 3: eigenvector for $\lambda=3$. Why? Substitute into $(A-\lambda I)v=0$ to find the direction. $\begin{pmatrix} -1 & 1 \\ 1 & -1 \end{pmatrix}\begin{pmatrix}x\\y\end{pmatrix}=0 \Rightarrow -x+y=0 \Rightarrow y=x$ Eigenvector $v_1 = \begin{pmatrix}1\\1\end{pmatrix}$ (any scale).

Step 4: eigenvector for $\lambda=1$. $\begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}v=0 \Rightarrow x+y=0 \Rightarrow y=-x$ Eigenvector $v_2 = \begin{pmatrix}1\\-1\end{pmatrix}$.

Check: $A\begin{pmatrix}1\\1\end{pmatrix} = \begin{pmatrix}3\\3\end{pmatrix} = 3\begin{pmatrix}1\\1\end{pmatrix}.$ ✓

Worked Example 2 — a triangular shortcut

$B = \begin{pmatrix} 4 & 2 \\ 0 & 5 \end{pmatrix}$

Step 1: $\det(B-\lambda I) = (4-\lambda)(5-\lambda) - (2)(0) = (4-\lambda)(5-\lambda)$. Why zero term? The lower-left is $0$, so the cross product vanishes.

Step 2: Roots are $\lambda = 4, 5$ — exactly the diagonal entries.

Two invariants that catch mistakes fast

WHY (sketch): The characteristic polynomial $\det(A-\lambda I) = (-1)^n(\lambda-\lambda_1)\cdots(\lambda-\lambda_n)$. Matching the $\lambda^{n-1}$ coefficient gives the sum = trace; the constant term ($\lambda=0$) gives $\det(A)=\prod \lambda_i$.

Example 1 check: trace $=2+2=4=1+3$ ✓, $\det = 4-1=3=1\times 3$ ✓.

Forecast-then-Verify

Common Mistakes (Steel-manned)

Flashcards

Connections

• Determinants — the tool that detects singularity ($\det=0$).
• Matrix as Linear Transformation — eigenvectors = directions not rotated.
• Diagonalization — $A = P\Lambda P^{-1}$ using eigenvectors as columns of $P$.
• Principal Component Analysis (PCA) — eigenvectors of the covariance matrix.
• Symmetric Matrices and Spectral Theorem — real orthogonal eigenvectors.
• Trace and Determinant — invariants equal to sum/product of eigenvalues.
• Null Space — eigenvectors live in the null space of $(A-\lambda I)$.

Concept Map

Hinglish (regional understanding)