Eigendecomposition of matrices — AI-ML notes

1. WHAT is an eigenvector / eigenvalue?

WHY the " $v \neq 0$ " rule? If we allowed $v=0$ , then $A\cdot 0 = \lambda \cdot 0$ holds for every $\lambda$ — useless. We forbid the zero vector so the eigenvalue is meaningful.

2. HOW to find eigenvalues — derive from scratch

Start from the defining equation and rearrange (do NOT memorize the char. polynomial — build it):

$A v = \lambda v \;\Longrightarrow\; Av - \lambda v = 0 \;\Longrightarrow\; (A - \lambda I)v = 0$

Now $(A-\lambda I)v = 0$ is a homogeneous system. We need a nonzero solution $v$ . A square system $Mv=0$ has a nonzero solution iff $M$ is singular, i.e.

$\boxed{\det(A - \lambda I) = 0}$

Once you have each $\lambda_i$ , plug it back and solve $(A-\lambda_i I)v = 0$ for the eigenvector(s) (the null space).

3. HOW to assemble the decomposition

Suppose $A$ (size $n$ ) has $n$ linearly independent eigenvectors $v_1,\dots,v_n$ with eigenvalues $\lambda_1,\dots,\lambda_n$ . Stack the eigenvectors as columns of $V$ and eigenvalues on the diagonal of $\Lambda$ :

$V = [\,v_1 \; v_2 \; \cdots \; v_n\,], \qquad \Lambda = \operatorname{diag}(\lambda_1,\dots,\lambda_n)$

Look at what $A$ does to all columns at once: $A V = [\,Av_1 \; \cdots \; Av_n\,] = [\,\lambda_1 v_1 \; \cdots \; \lambda_n v_n\,] = V\Lambda$

So $AV = V\Lambda$ . If $V$ is invertible (needs independent eigenvectors), right-multiply... actually multiply by $V^{-1}$ :

4. WHY this is powerful — matrix powers

$A^2 = (V\Lambda V^{-1})(V\Lambda V^{-1}) = V\Lambda \underbrace{V^{-1}V}_{I}\Lambda V^{-1} = V\Lambda^2 V^{-1}$

By induction: $\boxed{A^k = V\Lambda^k V^{-1}}, \qquad \Lambda^k = \operatorname{diag}(\lambda_1^k,\dots,\lambda_n^k)$

WHY huge: raising a diagonal matrix to a power is just powering the diagonal entries — $O(n)$ instead of repeated $O(n^3)$ multiplies. This is how Markov chains find steady states and how we analyze long-term behavior.

5. Special case: symmetric matrices

6. Worked examples

Worked example Example 1 — a

$2\times2$ diagonalizable matrix $A = \begin{pmatrix} 2 & 1 \\ 1 & 2\end{pmatrix}$

Step 1: Characteristic equation. Why? Eigenvalues are roots of $\det(A-\lambda I)$ . $\det\begin{pmatrix}2-\lambda & 1\\ 1 & 2-\lambda\end{pmatrix} = (2-\lambda)^2 - 1 = 0$ Step 2: Solve. Why? Roots give the scale factors. $(2-\lambda)^2 = 1 \Rightarrow 2-\lambda = \pm1 \Rightarrow \lambda = 1,\,3.$ Step 3: Eigenvector for $\lambda=3$ . Why? Solve $(A-3I)v=0$ to get the invariant direction. $\begin{pmatrix}-1&1\\1&-1\end{pmatrix}v=0 \Rightarrow v_1=v_2 \Rightarrow v=\frac{1}{\sqrt2}(1,1)$ . Step 4: Eigenvector for $\lambda=1$ . $\begin{pmatrix}1&1\\1&1\end{pmatrix}v=0 \Rightarrow v_1=-v_2 \Rightarrow v=\frac{1}{\sqrt2}(1,-1)$ . Check: symmetric matrix → eigenvectors are orthogonal ✅ ( $\;(1,1)\cdot(1,-1)=0$ ).

7. Common mistakes (Steel-man them)

8. Recall

Recall Active recall — cover the answers

What equation defines an eigenpair? → $Av=\lambda v$ .
Why do we require $\det(A-\lambda I)=0$ ? → so $(A-\lambda I)v=0$ has a nonzero solution (matrix singular).
What must $V$ contain and satisfy? → columns = eigenvectors; must be linearly independent (invertible).
Formula for $A^k$ ? → $V\Lambda^k V^{-1}$ .
When is $V$ orthogonal? → when $A$ is symmetric (spectral theorem).

Recall Explain to a 12-year-old (Feynman)

Imagine a big rubber sheet. When you pull on it, most drawn arrows on it swing to point somewhere new. But a few magic arrows keep pointing the same way — they just get longer or shorter. Those magic arrows are eigenvectors, and "how much longer" is the eigenvalue. If you know all the magic arrows, you understand everything the pull does: just stretch along each magic arrow. Eigendecomposition is writing down that list of magic arrows and their stretch amounts.

9. Connections

Determinants — the $\det=0$ singularity test that produces eigenvalues.
Diagonalization — the condition ( $n$ independent eigenvectors) for eigendecomposition to exist.
Singular Value Decomposition (SVD) — the fallback that always exists, even for non-square/defective matrices.
Principal Component Analysis (PCA) — eigendecomposition of the covariance matrix.
Markov Chains — steady states via $A^k = V\Lambda^k V^{-1}$ .
Symmetric Matrices & Spectral Theorem — orthonormal eigenbasis.

Eigenvector defining equation

Av=\lambda v

(with

v\neq 0

);

A

scales

v

without rotating it.

Why do we insert

I

A-\lambda I

To make

\lambda v = \lambda I v

a matrix–vector product so it can be subtracted from

Av

Characteristic equation

\det(A-\lambda I)=0

; its roots are the eigenvalues.

Why

\det(A-\lambda I)=0

A homogeneous system

(A-\lambda I)v=0

has a nonzero solution iff the matrix is singular, i.e. determinant zero.

Eigendecomposition formula

A=V\Lambda V^{-1}

V

= eigenvector columns,

\Lambda

= diag of eigenvalues.

Derive

AV=V\Lambda

A[v_1\dots v_n]=[\lambda_1 v_1\dots\lambda_n v_n]=V\Lambda

; then

A=V\Lambda V^{-1}

Formula for

A^k

A^k=V\Lambda^k V^{-1}

with

\Lambda^k=\mathrm{diag}(\lambda_i^k)

When is eigendecomposition impossible?

When

A

lacks

n

linearly independent eigenvectors (defective matrix), so

V

is not invertible.

Spectral theorem

Symmetric

A=A^\top

⇒ real eigenvalues, orthonormal eigenvectors,

A=Q\Lambda Q^\top

Are eigenvectors unique?

No — any nonzero scalar multiple works; we normalize to

\|v\|=1

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek matrix $A$ basically ek machine hai jo vectors ko transform karti hai — kisi ko ghumati hai, kisi ko stretch karti hai. Lekin kuch special vectors hote hain jinko $A$ sirf lamba ya chhota karti hai, direction bilkul same rehti hai. Yehi hote hain eigenvectors, aur kitna stretch hua woh number hai eigenvalue $\lambda$ . Equation simple hai: $Av=\lambda v$ .

Ab eigenvalue nikalne ke liye hum $Av=\lambda v$ ko rearrange karte hain: $(A-\lambda I)v=0$ . Yahan $I$ isliye ghusaya kyunki $\lambda$ ek scalar hai aur usko matrix banana zaroori tha subtract karne ke liye. Iss system ka nonzero solution tabhi milega jab matrix singular ho, matlab $\det(A-\lambda I)=0$ . Iss equation ke roots hi eigenvalues hain. Phir har $\lambda$ ko wapas daal ke null space solve karo — woh eigenvector hai.

Jab tumhare paas $n$ independent eigenvectors ho, unhe columns me rakh do $V$ me, aur eigenvalues ko diagonal $\Lambda$ me. Toh $A = V\Lambda V^{-1}$ . Isko yaad rakho "Enter, Stretch, Exit" — pehle eigen-coordinates me jao ( $V^{-1}$ ), phir har axis ko stretch karo ( $\Lambda$ ), phir wapas aao ( $V$ ). Sabse bada faayda: $A^k = V\Lambda^k V^{-1}$ , matlab power sirf diagonal pe lagao, bahut fast!

Yeh cheez ML me har jagah hai — PCA me covariance matrix (jo symmetric hoti hai) ka eigendecomposition hi principal directions deta hai. Ek warning: har matrix decompose nahi hoti — agar independent eigenvectors kam pad jaayein (defective matrix), toh SVD use karna padta hai. Bas yeh dhyaan rakho aur tum set ho.

Test yourself — Linear Algebra Essentials

Connections