Trace operator and properties — AI-ML notes

WHAT is the trace?

WHY does such a simple sum matter? Because it turns out to equal the sum of the eigenvalues and it behaves beautifully under products (cyclic property). In ML it appears everywhere: the trace of a covariance matrix is total variance, $\operatorname{tr}(A^\top A)$ is the squared Frobenius norm, and traces let us rewrite ugly quadratic forms into differentiable expressions.

Deriving the key properties FROM SCRATCH

We use only the definition $\operatorname{tr}(A)=\sum_i A_{ii}$ .

1. Linearity

$\operatorname{tr}(A+B)=\sum_i (A+B)_{ii}=\sum_i (A_{ii}+B_{ii}) = \operatorname{tr}(A)+\operatorname{tr}(B)$ $\operatorname{tr}(cA)=\sum_i cA_{ii}=c\sum_i A_{ii}=c\,\operatorname{tr}(A)$ Why this step? We just split a sum of sums — pure definition, no magic.

2. Transpose invariance

$\operatorname{tr}(A^\top)=\sum_i (A^\top)_{ii}=\sum_i A_{ii}=\operatorname{tr}(A)$ Why? The diagonal entries $A_{ii}$ are exactly the ones unchanged by transposing ( $(A^\top)_{ii}=A_{ii}$ ).

3. Cyclic property (the star result)

Start with two matrices where $A$ is $m\times n$ and $B$ is $n\times m$ (so both $AB$ and $BA$ are square): $\operatorname{tr}(AB)=\sum_{i=1}^{m}(AB)_{ii}=\sum_{i=1}^{m}\sum_{j=1}^{n}A_{ij}B_{ji}$ Why this step? Definition of trace, then definition of matrix product $(AB)_{ii}=\sum_j A_{ij}B_{ji}$ .

Now swap the order of summation (finite sums, always legal) and relabel: $=\sum_{j=1}^{n}\sum_{i=1}^{m}B_{ji}A_{ij}=\sum_{j=1}^{n}(BA)_{jj}=\operatorname{tr}(BA)$ Why this step? $\sum_i B_{ji}A_{ij}=(BA)_{jj}$ by the product definition. So

4. Trace = sum of eigenvalues

The characteristic polynomial is $p(\lambda)=\det(A-\lambda I)=\prod_i(\lambda_i-\lambda)$ . Expanding $\det(A-\lambda I)$ , the coefficient of $(-\lambda)^{n-1}$ is $\sum_i A_{ii}=\operatorname{tr}(A)$ . From the factored form that same coefficient is $\sum_i \lambda_i$ . Matching:

WHY invariant under similarity? If $A'=P^{-1}AP$ (a change of basis), then by cyclic property $\operatorname{tr}(P^{-1}AP)=\operatorname{tr}(APP^{-1})=\operatorname{tr}(A).$ So trace depends only on the map, not the chosen coordinates. That's the deep reason it equals a sum of eigenvalues.

5. Frobenius norm link

$\|A\|_F^2=\sum_{i,j}A_{ij}^2=\operatorname{tr}(A^\top A)$ Why? $(A^\top A)_{jj}=\sum_i A_{ij}A_{ij}=\sum_i A_{ij}^2$ ; summing over $j$ gives all entries squared.

Worked examples

Steel-man the mistakes

Forecast-then-Verify

Recall Predict before revealing

Let $A=\begin{pmatrix}0&1\\0&0\end{pmatrix}$ . Forecast $\operatorname{tr}(A)$ and its eigenvalues. Answer: $\operatorname{tr}(A)=0+0=0$ . Eigenvalues both $0$ , sum $=0$ . ✓ Note: a nonzero matrix can have zero trace.

Recall Feynman: explain to a 12-year-old

Imagine a square grid of numbers. Walk from the top-left corner straight down to the bottom-right corner — that diagonal line. Add up only the numbers you step on. That total is the "trace." The cool magic trick: if you tilt your head and look at the same shape from a different angle (change coordinates), that diagonal total stays exactly the same. It's like the weight of an object: doesn't change if you turn the object around.

Flashcards

What is the trace of a square matrix?

The sum of its main-diagonal entries,

\operatorname{tr}(A)=\sum_i A_{ii}

For which matrices is the trace defined?

Only square matrices.

State the cyclic property of trace.

\operatorname{tr}(AB)=\operatorname{tr}(BA)

, and

\operatorname{tr}(ABC)=\operatorname{tr}(BCA)=\operatorname{tr}(CAB)

Why is

\operatorname{tr}(A^\top)=\operatorname{tr}(A)

Transpose leaves diagonal entries unchanged:

(A^\top)_{ii}=A_{ii}

Trace equals the sum of what spectral quantity?

The eigenvalues,

\operatorname{tr}(A)=\sum_i\lambda_i

Why is trace invariant under similarity

P^{-1}AP

By cyclic property

\operatorname{tr}(P^{-1}AP)=\operatorname{tr}(APP^{-1})=\operatorname{tr}(A)

Express the Frobenius norm using trace.

\|A\|_F^2=\operatorname{tr}(A^\top A)

What does

\operatorname{tr}(xy^\top)

equal for column vectors?

The inner product

y^\top x = x^\top y

\operatorname{tr}(AB)=\operatorname{tr}(A)\operatorname{tr}(B)

No — trace is not multiplicative (counterexample

A=B=I_2

Meaning of

\operatorname{tr}(\Sigma)

for a covariance matrix?

Total variance = sum of variances of all features.

Connections

Eigenvalues and Eigenvectors — trace = sum of eigenvalues.
Determinant — determinant = product of eigenvalues (the multiplicative sibling).
Frobenius Norm — $\|A\|_F^2=\operatorname{tr}(A^\top A)$ .
Covariance Matrix and PCA — trace as total variance.
Matrix Calculus — trace trick for derivatives of quadratic forms.
Change of Basis and Similarity — trace as an invariant.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Trace ka matlab bahut simple hai: kisi bhi square matrix ke main diagonal (top-left se bottom-right) ke numbers ko add kar do — bas wahi trace hai, ek single number. Yaad rakho, sirf square matrices ka trace hota hai, rectangular ka nahi. Off-diagonal numbers ko ignore kar dete hain.

Ab isme khaas baat kya hai? Trace ka number coordinates badalne par nahi badalta. Matlab agar aap basis rotate kar do (similarity transform $P^{-1}AP$ ), trace same rehta hai. Iska deep reason yeh hai ki trace actually eigenvalues ka sum ke barabar hai — aur eigenvalues to map ki apni property hain, aapke likhe numbers ki nahi. Isliye trace ko "invariant" kehte hain, jaise kisi cheez ka weight ghumane se nahi badalta.

Sabse useful trick hai cyclic property: $\operatorname{tr}(AB)=\operatorname{tr}(BA)$ , aur teen matrices ho to $ABC\to BCA\to CAB$ — cycle (loop) ghuma sakte ho, lekin randomly reorder nahi. Ek dum common galti: log sochte hain $ABC$ aur $ACB$ same honge — nahi! Sirf cyclic rotation allowed hai.

ML me trace har jagah aata hai: covariance matrix ka trace = total variance (PCA ka base yahi hai), aur $\operatorname{tr}(A^\top A)=\|A\|_F^2$ yaani Frobenius norm squared. Ek aur pyaari cheez: do vectors ka outer product $xy^\top$ ka trace bas inner product $y^\top x$ ban jaata hai — isse loss functions ke derivatives nikalne me bahut aasaani hoti hai. Toh trace ko halke me mat lo, chhota dikhta hai par power-packed hai.

Test yourself — Linear Algebra Essentials

Connections