Covariance and correlation — Hinglish notes

1.3.9 · AI-ML › Probability & Statistics

Overview

Covariance aur correlation measure karte hain ki do random variables saath-saath kaise change karte hain. Ye answer karte hain: "Jab $X$ upar jaata hai, toh kya $Y$ bhi upar jaata hai (positive), neeche jaata hai (negative), ya unrelated rehta hai (zero)?" Ye concepts machine learning mein feature relationships, dimensionality reduction (PCA), aur regression samajhne ke liye foundational hain.

Core Intuition

Definitions and Derivations

Computational Form Ki Derivation

Definition se shuru karo: $Cov (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})]$

Product expand karo: $= E [X Y - X μ_{Y} - μ_{X} Y + μ_{X} μ_{Y}]$

Expectation ki linearity apply karo (expectation addition par distribute ho jaati hai): $= E [X Y] - E [X μ_{Y}] - E [μ_{X} Y] + E [μ_{X} μ_{Y}]$

Ye step kyun? Constants ko expectations ke bahar nikaala ja sakta hai: $= E [X Y] - μ_{Y} E [X] - μ_{X} E [Y] + μ_{X} μ_{Y}$

Substitute karo $μ_{X} = E [X]$ aur $μ_{Y} = E [Y]$ : $= E [X Y] - μ_{Y} μ_{X} - μ_{X} μ_{Y} + μ_{X} μ_{Y}$

Simplify karo (beech ke do terms cancel hokar ek reh jaata hai): $Cov (X, Y) = E [X Y] - E [X] E [Y]$

Interpretation:

Agar $X$ aur $Y$ independent hain, toh $E [X Y] = E [X] E [Y]$ , isliye $Cov (X, Y) = 0$ .
Positive covariance: $X$ aur $Y$ saath-saath increase karte hain.
Negative covariance: jab $X$ increase karta hai, $Y$ decrease karta hai.
Units: covariance ke units $X \times Y$ hote hain (jaise meters × kilograms), jisse magnitude interpret karna mushkil ho jaata hai.

Standard Deviations Se Normalize Kyun Karte Hain?

Intuition: Covariance $X$ aur $Y$ ke scales par depend karta hai. Agar hum height ko meters ki jagah millimeters mein measure karein, toh covariance $100 0^{2}$ se scale ho jaata hai! $σ_{X} σ_{Y}$ se divide karne par ye scale dependence hat jaati hai.

Bounds ki derivation ( $- 1 \leq ρ \leq 1$ ):

Standardized variables consider karo: $Z_{X} = \frac{X - μ _{X}}{σ _{X}}, Z_{Y} = \frac{Y - μ _{Y}}{σ _{Y}}$

Inका mean $0$ aur variance $1$ hota hai. Correlation hai: $ρ = E [Z_{X} Z_{Y}]$

Cauchy-Schwarz inequality kehti hai: $∣ E [Z_{X} Z_{Y}] ∣ \leq E [Z_{X}^{2}] E [Z_{Y}^{2}] = 1 \cdot 1 = 1$

Ye step kyun? $E [Z_{X}^{2}] = Var (Z_{X}) = 1$ standardization ki wajah se.

Isliye $- 1 \leq ρ \leq 1$ . Equality tab hold hoti hai jab $Z_{Y} = \pm Z_{X}$ (perfect linear relationship).

Sample Estimates (Finite Data)

Practice mein, hum $n$ samples $(x_{1}, y_{1}), \dots, (x_{n}, y_{n})$ se estimate karte hain:

Visual Understanding

Diagram alag-alag correlation values ke liye scatter plots dikhata hai. Notice karo ki:

$ρ = 0$ : bina kisi trend ke cloud
$∣ ρ ∣ = 0.5$ : moderate trend
$∣ ρ ∣ = 1$ : points bilkul ek line par

Worked Examples

Worked example Example 1: Haath Se Covariance Compute Karna

Diya gaya joint distribution:

$X$	$Y$	$P (X, Y)$
1	2	0.2
1	4	0.3
3	2	0.1
3	4	0.4
Step 1: Marginal means compute karo
$E [X] = 1 (0.2 + 0.3) + 3 (0.1 + 0.4) = 1 (0.5) + 3 (0.5) = 2$
$E [Y] = 2 (0.2 + 0.1) + 4 (0.3 + 0.4) = 2 (0.3) + 4 (0.7) = 3.4$
Ye step kyun? Marginal expectation har variable ke liye saare joint outcomes par sum karta hai.

Step 2: $E [X Y]$ compute karo $E [X Y] = (1 \cdot 2) (0.2) + (1 \cdot 4) (0.3) + (3 \cdot 2) (0.1) + (3 \cdot 4) (0.4)$ $= 0.4 + 1.2 + 0.6 + 4.8 = 7$

Ye step kyun? Function $g (X, Y)$ ki expectation $\sum g (x, y) P (x, y)$ hoti hai. Step 3: Formula apply karo $Cov (X, Y) = 7 - (2) (3.4) = 7 - 6.8 = 0.2$

Interpretation: Positive covariance matlab $X$ aur $Y$ saath-saath increase karte hain (thoda kam strong).

Worked example Example 2: Data Se Sample Correlation

Data points: $(1, 2), (2, 4), (3, 5), (4, 7)$

Step 1: Means compute karo $\overset{x}{ˉ} = \frac{1 + 2 + 3 + 4}{4} = 2.5, \overset{y}{ˉ} = \frac{2 + 4 + 5 + 7}{4} = 4.5$

Step 2: Deviations

$x_{i}$	$y_{i}$	$x_{i} - \overset{x}{ˉ}$	$y_{i} - \overset{y}{ˉ}$	$(x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ})$	$(x_{i} - \overset{x}{ˉ})^{2}$	$(y_{i} - \overset{y}{ˉ})^{2}$
1	2	-1.5	-2.5	3.75	2.25	6.25
2	4	-0.5	-0.5	0.25	0.25	0.25
3	5	0.5	0.5	0.25	0.25	0.25
4	7	1.5	2.5	3.75	2.25	6.25

Sums: $\sum (x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ}) = 8$ , $\sum (x_{i} - \overset{x}{ˉ})^{2} = 5$ , $\sum (y_{i} - \overset{y}{ˉ})^{2} = 13$

Ye step kyun? Hum numerator aur denominator ke pieces alag-alag compute kar rahe hain.

Step 3: Sample correlation $r = \frac{8}{5 13} = \frac{8}{65} = \frac{8}{8.062} \approx 0.992$

Interpretation: Bahut strong positive linear relationship (almost perfect).

Common Mistakes

Key Properties and Facts

Machine Learning Context

ML mein covariance/correlation kyun matter karta hai:

Feature Engineering: Highly correlated features redundant ho sakte hain (regression mein multicollinearity).
PCA (Principal Component Analysis): Maximum variance ki directions dhundhta hai = covariance matrix ke eigenvectors.
Correlation-based Feature Selection: Target ke saath $∣ ρ ∣ < ϵ$ wale features remove karo.
Gaussian Processes: Covariance functions (kernels) data points ke beech similarity define karte hain.
Portfolio Optimization: Finance ML mein, covariance matrices asset co-movements model karti hain.

Feynman Technique (ELI12)

Recall Aise Samjhao Jaise Main 12 Saal Ka Hoon

Socho tum aur tumhara dost dono skateboard seekh rahe ho. Har hafte, tum dono harder tricks try karte ho.

Covariance ye poochne jaisa hai: "Jab main harder trick seekhta hoon, kya mere dost us hafte bhi harder trick seekhte hain?" Agar haan, covariance positive hai. Agar jab main better karta hoon toh wo worse karte hain, toh negative hai. Agar koi pattern nahi hai, toh zero hai. Lekin baat ye hai: agar main apni progress "tricks learned" mein measure karoon aur wo "hours practiced" mein, toh numbers messy ho jaate hain. Maan lo maine 5 tricks sikhe, unhone 20 ghante practice kiye—hum compare kaise karein?

Correlation ye fix karta hai sab kuch ek hi scale -1 se +1 par rakhke. Ye hamari teamwork ko rate karne jaisa hai: +1 matlab hum perfectly sync mein hain (jab main improve karta hoon, wo bhi apni respective ranges ke relative exact same amount improve karte hain), -1 matlab hum perfect opposites hain, aur 0 matlab hum apna-apna kaam kar rahe hain bina kisi pattern ke.

Toh: covariance = raw teamwork, correlation = report card scale par teamwork.

Mnemonic

Active Recall Flashcards

#flashcards/ai-ml

Random variables

X

aur

Y

ke beech covariance ki definition kya hai?

Cov (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})] = E [X Y] - E [X] E [Y]

Pearson correlation coefficient ka formula kya hai?

ρ_{X, Y} = \frac{Cov ( X , Y )}{σ _{X} σ _{Y}}

, jo covariance ko

[- 1, 1]

mein normalize karta hai

Correlation

ρ

ki range kya hai?

- 1 \leq ρ \leq 1

(dimensionless, Cauchy-Schwarz inequality se)

Agar

X

aur

Y

independent hain, toh

Cov (X, Y)

kya hai?

Cov (X, Y) = 0

(lekin converse hamesha true nahi hota!)

Kya

Cov (X, Y) = 0

imply karta hai ki

X

aur

Y

independent hain?

Nahi! Zero covariance sirf linear relationships ko rule out karta hai. Example:

Y = X^{2}

jab

X

zero ke around symmetric ho toh zero covariance hai lekin complete dependence hai.

Var (X + Y)

ka formula kya hai?

Var (X + Y) = Var (X) + Var (Y) + 2 Cov (X, Y)

Sample covariance mein

n

ki jagah

n - 1

kyun use karte hain?

Bessel's correction: sample means compute karte waqt hum ek degree of freedom khote hain, jisse estimator unbiased ho jaata hai.

ρ = + 1

ka matlab kya hai?

Perfect positive linear relationship:

Y = a X + b

jab

a > 0

ρ = - 1

ka matlab kya hai?

Perfect negative linear relationship:

Y = a X + b

jab

a < 0

Relationship strength compare karne ke liye correlation, covariance se better kyun hai?

Correlation scale-invariant (dimensionless) hai, jabki covariance

X

aur

Y

ke units par depend karta hai.

Covariance ka ek ML application batao.

PCA (Principal Component Analysis) maximum variance ki directions dhundhne ke liye covariance matrix use karta hai.

Cov (a X + b, Y)

kiske barabar hai?

a Cov (X, Y)

(constants

b

hat jaate hain;

a

se scale karne par covariance scale hoti hai)

Connections

1.3.01-Random-variables — covariance, variance ko RV pairs tak extend karta hai
1.3.05-Expectation-and-variance — covariance expectation operator use karta hai
1.3.08-Joint-and-marginal-distributions — covariance joint distributions se compute hota hai
2.4.01-Principal-component-analysis — PCA covariance matrix ko diagonalize karta hai
3.2.05-Linear-regression — correlation linear relationship ki strength measure karta hai
3.2.06-Multicolinearity — high inter-feature correlation issues cause karta hai
1.2.04-Independence — independent RVs ka zero covariance hota hai
4.1.03-Feature-selection — feature ranking ke liye correlation-based methods

In concepts ko master karo, aur tum samajh jaoge ki variables kaise saath dance karte hain—theory se lekar ML applications tak.