Visual walkthrough — Covariance and correlation

4.9.12 · D2 · Maths › Probability Theory & Statistics › Covariance and correlation

Hum ek hi object par measure kiye gaye do numbers ke saath kaam karte hain — jaise kisi insaan ki height $X$ aur weight $Y$. Har insaan ek dot hai graph par: horizontal position $=X$, vertical position $=Y$.

Step 1 — Dots ka cloud plot karo

KYA. Hum kaafi saare $(X,Y)$ pairs collect karte hain aur har ek ko dot ki tarah drop karte hain. Yeh picture ek scatter plot hai.

KYUN. Kisi bhi formula se pehle, hume raw question dekhna hai: "jab $X$ bada hota hai, toh kya $Y$ bhi bada hota hai?" Ek cloud jo up-right slope karta hai woh haan kehta hai; neeche slope karta hua cloud opposite kehta hai; ek shapeless blob kehta hai koi pattern nahi. Neeche sab kuch bas yeh visual slope ko ek number mein badle ka tarika hai.

PICTURE. Neeche blue cloud ka up-right tilt dekho.

Step 2 — Har variable ka balance point $\mu$ dhundho

KYA. ${\mu }_X=E[X]$ aur ${\mu }_Y=E[Y]$ compute karo — saare horizontal positions ka average aur saare vertical positions ka average. Do lines $X={\mu }_X$ (vertical) aur $Y={\mu }_Y$ (horizontal) khiincho.

KYUN. "High" aur "low" meaningless hain jab tak hum ek reference nahi choose karte. Mean sabse fair reference hai: yeh woh point hai jiske around dots balance karte hain. Yeh do lines plane ko char rooms (quadrants) mein kaatti hain jinhe hum agle step mein signs se label karenge.

PICTURE. Orange crosshair pair $({\mu }_X,{\mu }_Y)$ hai — cloud ka centre of mass.

$\underset{\text{avg }X\text{, the vertical line}}{\underbrace{{\mu }_X}}=E[X],\underset{\text{avg }Y\text{, the horizontal line}}{\underbrace{{\mu }_Y}}=E[Y]$

Step 3 — Har dot ko centre se deviation ki tarah measure karo

KYA. $(X,Y)$ par ek dot ke liye, uske do deviations banao: $\underset{\text{how far right }(+)\text{ or left }(-)}{\underbrace{X-{\mu }_X}},\underset{\text{how far up }(+)\text{ or down }(-)}{\underbrace{Y-{\mu }_Y}}.$

KYUN. Absolute positions humein "agreement" nahi batate. Centre ke relative batate hain: centre ke right wala dot $X-{\mu }_X>0$ rakhta hai; left wala, $<0$. Vertically bhi yehi idea. Yeh exactly Variance and Standard Deviation ka raw material hai — wahan humne ek deviation ko square kiya; yahan hum do ko pair karenge.

PICTURE. Green arrow horizontal deviation hai, red arrow vertical deviation, orange centre se dot tak measured.

Step 4 — Do deviations ko multiply karo: har room ke liye ek sign

KYA. Har dot ke liye product $(X-{\mu }_X)(Y-{\mu }_Y)$ compute karo.

KYUN. Hum ek aisa operation chahte hain jo agreement ke liye +, disagreement ke liye − kahe. Multiplication sabse sasta aisa operation hai, kyunki "same sign × same sign = +" aur "opposite signs = −". Char rooms walk karo:

• Top-right (right aur up): $(+)(+)=+$ — agree karte hain (dono average se upar).
• Bottom-left (left aur down): $(-)(-)=+$ — agree karte hain (dono average se neeche).
• Top-left (left, up): $(-)(+)=-$ — disagreement.
• Bottom-right (right, down): $(+)(-)=-$ — disagreement.

Toh product ka sign ek vote hai: $+$ vote "saath move karne" ke liye, $-$ vote "alag move karne" ke liye. Iska size batata hai ki us dot ka vote kitna emphatic hai (ek dot jo kisi corner mein bahut door hai woh loudly vote karta hai).

PICTURE. Har room ko product ke sign ke hisaab se shade kiya gaya hai; agreement rooms green $(+)$, disagreement rooms red $(-)$.

Step 5 — Votes average karo: yahi covariance hai

KYA. Un saare signed products ki expectation (average) lo: $\mathrm{Cov}(X,Y)=E\left[\underset{\text{horiz. deviation}}{\underbrace{(X-{\mu }_X)}}\underset{\text{vert. deviation}}{\underbrace{(Y-{\mu }_Y)}}\right].$

KYUN. Averaging se green $(+)$ votes aur red $(-)$ votes cancel ya dominate hote hain. Net positive ⇒ cloud up-right lean karta hai (saath move karte hain). Net negative ⇒ down-right lean karta hai. Near zero ⇒ koi linear tilt nahi. Yeh ek number Step 1 ki picture ki poori kahani hai.

PICTURE. Teen clouds side by side: positive covariance (up-tilt), zero (round blob), negative (down-tilt) — har cloud ka net vote dikhaya gaya hai.

Step 6 — Units ki problem: covariance strength kyun measure nahi kar sakta

KYA. Poore cloud ko horizontally factor $10$ se stretch karo (height millimetres mein measure karo centimetres ki bajay). Tilt waise hi dikhta hai, lekin covariance number $10$ se multiply ho jaata hai.

KYUN. Bilinearity se, $\mathrm{Cov}(aX,Y)=a\mathrm{Cov}(X,Y)$. Toh ek bada number zaroori nahi ki zyada strong relationship ho — ho sakta hai bas bade units hon. Hume har variable ka apna spread divide out karna hoga.

PICTURE. Same-shaped cloud, x-axis relabelled — covariance $0.15$ se $1.5$ tak jump karta hai bina pattern mein koi change ke.

Step 7 — Covariance ko leash par rakho: correlation $\rho$

KYA. Define karo ${\rho }_{XY}=\frac{\mathrm{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}.$

KYUN. ${\sigma }_X$ se divide karne se horizontal spread $1$ ho jaata hai, aur ${\sigma }_Y$ se vertical spread $1$. Stretched aur unstretched clouds ab identical dikhte hain — aur same $\rho$ dete hain. Toh $\rho$ ek unitless score hai sirf pattern shape ka, Step 6 ke trick se immune.

PICTURE. Har axis ko standardise karne ke baad, cloud ek box ke andar fit ho jaata hai; best-fit line ka tilt woh hai jo $\rho$ report karta hai.

Step 8 — $\rho$ kabhi $[-1,1]$ se kyun nahi jaata (leash proof, dekha gaya)

KYA. $U=X-{\mu }_X$, $V=Y-{\mu }_Y$ lo. Kisi bhi real slope $t$ ke liye, quantity $E[(U-tV{)}^2]$ squares ka average hai, isliye yeh $\ge 0$ hai: $\underset{={\sigma }_Y^2}{\underbrace{E[V^2]}}{t}^2-\underset{=2\mathrm{Cov}}{\underbrace{2E[UV]}}t+\underset{={\sigma }_X^2}{\underbrace{E[U^2]}}\ge 0.$

KYUN. Ek quadratic in $t$ jo kabhi $0$ se neeche nahi jaata woh axis ko zyada se zyada ek baar touch kar sakta hai — iska discriminant $\le 0$ hona chahiye: $(2\mathrm{Cov}{)}^2-4{\sigma }_Y^2{\sigma }_X^2\le 0⟹{\mathrm{Cov}}^2\le {\sigma }_X^2{\sigma }_Y^2⟹|\rho |\le 1.$ Yeh exactly Cauchy–Schwarz Inequality hai. Equality ($\rho =\pm 1$) tabhi hoti hai jab koi $t$ $E[(U-tV{)}^2]=0$ bana de, yaani $U=tV$ bina kisi scatter ke — $Y$ $X$ mein ek exactly straight line hai.

PICTURE. Parabola $f(t)={\sigma }_Y^2{t}^2-2\mathrm{Cov}t+{\sigma }_X^2$ axis par ya uske upar baitha hai; jab woh axis ko sirf kiss karta hai, $\rho =\pm 1$.

Step 9 — Degenerate cases jinse tumhe kabhi surprise nahi hona chahiye

KYA & KYUN. Char edge scenarios, har ek mein $\rho$ kya karta hai:

• Perfect line, up: saare dots ek rising straight line par ⇒ $\rho =+1$ (Step 8 ki parabola $t>0$ par axis ko touch karti hai).
• Perfect line, down: falling straight line ⇒ $\rho =-1$.
• Flat / vertical cloud: agar $X$ kabhi vary nahi karta toh ${\sigma }_X=0$, aur $\rho =\frac{0}{0}$ undefined hai — report karne ke liye koi "tilt" nahi hai.
• Curved but symmetric, e.g. $Y=X^2$ with $X$ symmetric about $0$: green aur red votes exactly cancel ho jaate hain, isliye $\mathrm{Cov}=0$ aur $\rho =0$ even though $Y$ fully $X$ se determined hai. Covariance sirf relationship ka linear part dekhta hai. Isliye $\rho =0$ ka matlab Independence of Random Variables nahi hai.

PICTURE. Ek strip mein char cases, har ek apne $\rho$ ke saath labelled.

Ek picture mein poora summary

Sab kuch compressed: dots → mean se deviations → signed products (green agree, red disagree) → unhe average karo (covariance) → spreads se divide karo leashed score $\rho \in [-1,1]$ paane ke liye.

Active recall

Connections

• Covariance and correlation — parent topic jise yeh pictures unpack karti hain.
• Expectation of Random Variables — averaging machine $E[\cdot ]$ jo har step mein use hoti hai.
• Variance and Standard Deviation — deviations aur $\sigma$ iske building blocks hain.
• Cauchy–Schwarz Inequality — Step 8 ki leash.
• Independence of Random Variables — kyun $\rho =0$ independence se weaker hai.
• Linear Regression — Step 7 ki tilt line precisely banayi gayi.
• Covariance Matrix — saare pairwise covariances ek saath.