Intuition The big pictureA matrix A A A transformation that stretches, rotates, and shears space. Its inverse A − 1 A^{-1} A − 1 perfectly undoes A A A WHY it matters in ML: solving A x = b A\mathbf{x} = \mathbf{b} A x = b one unique solution vs. none or infinitely many . Invertibility is the mathematical name for "this transformation loses no information."
For a square n × n n\times n n × n A A A A − 1 A^{-1} A − 1 A A − 1 = A − 1 A = I n A A^{-1} = A^{-1} A = I_n A A − 1 = A − 1 A = I n I n I_n I n A A A invertible (or nonsingular ). If it does not exist, A A A singular .
WHY must A A A A − 1 A^{-1} A − 1 n n n n n n pseudo-inverse exists).
We don't memorize the formula — we build it. Let
A = [ a b c d ] , A − 1 = [ x y z w ] . A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}, \qquad A^{-1} = \begin{bmatrix} x & y \\ z & w \end{bmatrix}. A = [ a c b d ] , A − 1 = [ x z y w ] .
We demand A A − 1 = I AA^{-1} = I A A − 1 = I [ a b c d ] [ x y z w ] = [ 1 0 0 1 ] . \begin{bmatrix} a & b \\ c & d \end{bmatrix}\begin{bmatrix} x & y \\ z & w \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}. [ a c b d ] [ x z y w ] = [ 1 0 0 1 ] .
Why this step? The definition of inverse is exactly this equation — we just write it out and solve.
Column 1 gives: a x + b z = 1 ax + bz = 1 a x + b z = 1 c x + d z = 0 cx + dz = 0 c x + d z = 0 z = − c d x z = -\tfrac{c}{d}x z = − d c x a x − b c d x = 1 ⟹ x a d − b c d = 1 ⟹ x = d a d − b c . ax - \tfrac{bc}{d}x = 1 \implies x\frac{ad-bc}{d} = 1 \implies x = \frac{d}{ad-bc}. a x − d b c x = 1 ⟹ x d a d − b c = 1 ⟹ x = a d − b c d .
Why this step? We eliminate one unknown to isolate x x x a d − b c ad-bc a d − b c forces itself out — that is the quantity that must be nonzero to divide by.
Doing the same for all four unknowns:
The whole thing blows up if a d − b c = 0 ad-bc = 0 a d − b c = 0 determinant , and it is the gatekeeper of invertibility.
Intuition Determinant = signed volume scale factordet A \det A det A A A A det A = 0 \det A = 0 det A = 0 collapses space onto a lower dimension (a line, a plane). Once volume is crushed to zero, you can't recover it — infinitely many inputs map to the same output, so no unique undo exists . Hence A − 1 A^{-1} A − 1 iff det A ≠ 0 \det A \neq 0 det A = 0
WHY #7? If A v = 0 ⋅ v A\mathbf{v} = 0\cdot\mathbf{v} A v = 0 ⋅ v v ≠ 0 \mathbf{v}\neq 0 v = 0 0 \mathbf{0} 0 det A = ∏ λ i \det A = \prod \lambda_i det A = ∏ λ i det A = 0 \det A = 0 det A = 0
WHY it works: the identity A ⋅ adj ( A ) = ( det A ) I A\cdot\operatorname{adj}(A) = (\det A) I A ⋅ adj ( A ) = ( det A ) I det A \det A det A n ≥ 3 n\ge 3 n ≥ 3 Gauss–Jordan ([ A ∣ I ] → [ I ∣ A − 1 ] [A\,|\,I] \to [I\,|\,A^{-1}] [ A ∣ I ] → [ I ∣ A − 1 ]
Worked example Example 1 — a clean2 × 2 2\times2 2 × 2 A = [ 4 7 2 6 ] A = \begin{bmatrix} 4 & 7 \\ 2 & 6\end{bmatrix} A = [ 4 2 7 6 ] Step 1: det A = 4 ⋅ 6 − 7 ⋅ 2 = 24 − 14 = 10 \det A = 4\cdot 6 - 7\cdot 2 = 24 - 14 = 10 det A = 4 ⋅ 6 − 7 ⋅ 2 = 24 − 14 = 10 Why? Nonzero → invertible.
Step 2: Swap diagonal, negate off-diagonal: [ 6 − 7 − 2 4 ] \begin{bmatrix} 6 & -7 \\ -2 & 4\end{bmatrix} [ 6 − 2 − 7 4 ] Why? That's the adjugate for 2 × 2 2\times2 2 × 2 Step 3: Divide by 10 10 10 A − 1 = [ 0.6 − 0.7 − 0.2 0.4 ] A^{-1} = \begin{bmatrix} 0.6 & -0.7 \\ -0.2 & 0.4\end{bmatrix} A − 1 = [ 0.6 − 0.2 − 0.7 0.4 ] Check: A A − 1 = I A A^{-1} = I A A − 1 = I
Worked example Example 2 — a singular matrixB = [ 1 2 2 4 ] B = \begin{bmatrix} 1 & 2 \\ 2 & 4\end{bmatrix} B = [ 1 2 2 4 ] Step 1: det B = 1 ⋅ 4 − 2 ⋅ 2 = 0 \det B = 1\cdot4 - 2\cdot2 = 0 det B = 1 ⋅ 4 − 2 ⋅ 2 = 0 Why matters? Zero determinant → not invertible .
Observe: row 2 = 2 × 2\times 2 × = 1 < 2 =1<2 = 1 < 2 B x = 0 B\mathbf{x}=\mathbf{0} B x = 0 x = [ 2 − 1 ] \mathbf{x}=\begin{bmatrix}2\\-1\end{bmatrix} x = [ 2 − 1 ] [ 1 ⋅ 2 + 2 ( − 1 ) 2 ⋅ 2 + 4 ( − 1 ) ] = 0 \begin{bmatrix}1\cdot2+2(-1)\\2\cdot2+4(-1)\end{bmatrix}=\mathbf{0} [ 1 ⋅ 2 + 2 ( − 1 ) 2 ⋅ 2 + 4 ( − 1 ) ] = 0
Worked example Example 3 — solving via inverseSolve A x = b A\mathbf{x} = \mathbf{b} A x = b A A A b = [ 1 1 ] \mathbf{b}=\begin{bmatrix}1\\1\end{bmatrix} b = [ 1 1 ] x = A − 1 b = [ 0.6 − 0.7 − 0.2 0.4 ] [ 1 1 ] = [ − 0.1 0.2 ] \mathbf{x} = A^{-1}\mathbf{b} = \begin{bmatrix} 0.6 & -0.7 \\ -0.2 & 0.4\end{bmatrix}\begin{bmatrix}1\\1\end{bmatrix} = \begin{bmatrix}-0.1\\0.2\end{bmatrix} x = A − 1 b = [ 0.6 − 0.2 − 0.7 0.4 ] [ 1 1 ] = [ − 0.1 0.2 ] Why unique? Because det A ≠ 0 \det A\neq0 det A = 0 (In ML we rarely form A − 1 A^{-1} A − 1
( A B ) − 1 = A − 1 B − 1 (AB)^{-1} = A^{-1}B^{-1} ( A B ) − 1 = A − 1 B − 1 Why it feels right: distributing over products works for scalars and transposes-feel similar. The fix: inverses reverse order like putting on/taking off clothes:
( A B ) − 1 = B − 1 A − 1 . (AB)^{-1} = B^{-1}A^{-1}. ( A B ) − 1 = B − 1 A − 1 . ( A B ) ( B − 1 A − 1 ) = A ( B B − 1 ) A − 1 = A I A − 1 = I (AB)(B^{-1}A^{-1}) = A(BB^{-1})A^{-1} = AIA^{-1} = I ( A B ) ( B − 1 A − 1 ) = A ( B B − 1 ) A − 1 = A I A − 1 = I
Common mistake "Only square matrices matter, so any square matrix is invertible."Why it feels right: the inverse is only defined for square matrices, so "square ⇒ invertible" seems automatic. The fix: square is necessary but not sufficient . [ 1 2 2 4 ] \begin{bmatrix}1&2\\2&4\end{bmatrix} [ 1 2 2 4 ] det = 0 \det=0 det = 0 also need full rank / nonzero determinant.
Common mistake "A tiny nonzero determinant is fine."Why it feels right: technically det ≠ 0 \det\neq0 det = 0 The fix: numerically, a nearly-singular matrix (huge condition number ) amplifies errors wildly. In ML we add regularization (A + λ I A + \lambda I A + λ I
Recall Feynman: explain to a 12-year-oldImagine a machine that takes a photo and scrambles it in a fixed way. The inverse machine unscrambles it back to the original. But if the scrambler ever throws away part of the picture (like painting over half of it), no un-scrambler can bring it back — too much was lost. The determinant is a number that tells you "did the machine throw anything away?" If it's zero, information was lost and there's no undo button . If it's not zero, you can always undo it perfectly.
Mnemonic Remember the conditions"Dr. Rank Loves Nulls Uniquely, Not Eigen-zeros, Ident-ifiable."
D et≠0 · Rank full · L inearly independent cols · Null space trivial · U nique solutions · Not 0 eigenvalue · row-reduces to Ident ity. All rise or fall together.
What equation defines the inverse A − 1 A^{-1} A − 1 A A − 1 = A − 1 A = I AA^{-1} = A^{-1}A = I A A − 1 = A − 1 A = I .
For a 2 × 2 2\times2 2 × 2 1 a d − b c [ d − b − c a ] \frac{1}{ad-bc}\begin{bmatrix}d&-b\\-c&a\end{bmatrix} a d − b c 1 [ d − c − b a ] .
A matrix is invertible iff its determinant is…? nonzero (
det A ≠ 0 \det A \neq 0 det A = 0 ).
Name three conditions equivalent to invertibility. Full rank, linearly independent columns, trivial null space (also: unique solutions, 0 not an eigenvalue, row-reduces to
I I I ).
What is ( A B ) − 1 (AB)^{-1} ( A B ) − 1 B − 1 A − 1 B^{-1}A^{-1} B − 1 A − 1 (order reverses).
Why can't a non-square matrix have a true inverse? It changes dimension, so no single matrix maps outputs back one-to-one; only a pseudo-inverse exists.
If det A = 0 \det A = 0 det A = 0 Collapses space onto a lower dimension, losing information → no unique undo.
Relation between determinant and eigenvalues? det A = ∏ i λ i \det A = \prod_i \lambda_i det A = ∏ i λ i ; a zero eigenvalue ⇒
det = 0 \det = 0 det = 0 ⇒ singular.
Why is a tiny determinant risky numerically? Large condition number amplifies rounding errors; matrix is nearly singular (fix: regularize with
+ λ I +\lambda I + λ I ).
What does adj ( A ) \operatorname{adj}(A) adj ( A ) The transpose of the cofactor matrix;
A − 1 = 1 det A adj ( A ) A^{-1} = \frac{1}{\det A}\operatorname{adj}(A) A − 1 = d e t A 1 adj ( A ) .
Matrix A as transformation
Swap diag negate off divide by det
Unique solution to Ax = b
Linear regression normal equations
Intuition Hinglish mein samjhoSocho ek matrix A A A inverse A − 1 A^{-1} A − 1 A A − 1 = I AA^{-1}=I A A − 1 = I A x = b A\mathbf{x}=\mathbf{b} A x = b
Sabse important cheez hai determinant . Yeh number batata hai ki machine ne area/volume kitna scale kiya. Agar det A = 0 \det A = 0 det A = 0 collapse kar diya — information gum ho gayi, aur jab information gum ho jaye to undo karna impossible hai. Isliye rule: A − 1 A^{-1} A − 1 det A ≠ 0 \det A \neq 0 det A = 0 2 × 2 2\times2 2 × 2 a d − b c ad-bc a d − b c
Ek common galti: log sochte hain ( A B ) − 1 = A − 1 B − 1 (AB)^{-1}=A^{-1}B^{-1} ( A B ) − 1 = A − 1 B − 1 B − 1 A − 1 B^{-1}A^{-1} B − 1 A − 1
Practical tip: agar determinant bahut chhota (nearly zero) hai to numerically inverse bahut unstable ho jata hai — chhoti errors badi ban jati hain (high condition number). Isliye ML mein hum A + λ I A+\lambda I A + λ I
