These three concepts answer different questions about a random variable:
Expectation (Mean) : Where is the "center" of the distribution?Variance : How spread out are the values?Standard Deviation : How spread out, in the same units as the original data?
They form the foundation for understanding data distributions, comparing models, and quantifying uncertainty in ML.
Intuition The Long-Run AverageIf you repeated an experiment infinitely many times, expectation is the average value you'd get. It's the "center of mass" of a probability distribution—where the distribution would balance if it were a physical object.
Definition Mathematical DefinitionFor a discrete random variable X X X x i x_i x i P ( X = x i ) P(X = x_i) P ( X = x i ) E [ X ] = ∑ i x i ⋅ P ( X = x i ) E[X] = \sum_{i} x_i \cdot P(X = x_i) E [ X ] = ∑ i x i ⋅ P ( X = x i ) continuous random variable with PDF f ( x ) f(x) f ( x ) E [ X ] = ∫ − ∞ ∞ x ⋅ f ( x ) d x E[X] = \int_{-\infty}^{\infty} x \cdot f(x) \, dx E [ X ] = ∫ − ∞ ∞ x ⋅ f ( x ) d x
Notation: E [ X ] E[X] E [ X ] μ \mu μ ⟨ X ⟩ \langle X \rangle ⟨ X ⟩
Why this definition?
Imagine you run an experiment N N N x i x_i x i n i n_i n i Average = 1 N ∑ i n i ⋅ x i = ∑ i x i ⋅ n i N \text{Average} = \frac{1}{N} \sum_{i} n_i \cdot x_i = \sum_{i} x_i \cdot \frac{n_i}{N} Average = N 1 ∑ i n i ⋅ x i = ∑ i x i ⋅ N n i
As N → ∞ N \to \infty N → ∞ n i N → P ( X = x i ) \frac{n_i}{N} \to P(X = x_i) N n i → P ( X = x i ) lim N → ∞ Average = ∑ i x i ⋅ P ( X = x i ) = E [ X ] \lim_{N \to \infty} \text{Average} = \sum_{i} x_i \cdot P(X = x_i) = E[X] lim N → ∞ Average = ∑ i x i ⋅ P ( X = x i ) = E [ X ]
For continuous variables : We divide the real line into tiny bins of width d x dx d x x x x f ( x ) d x f(x) \, dx f ( x ) d x
Why linearity? Expand the sum:
E [ a X + b Y ] = ∑ i ∑ j ( a x i + b y j ) P ( X = x i , Y = y j ) E[aX + bY] = \sum_i \sum_j (ax_i + by_j) P(X = x_i, Y = y_j) E [ a X + bY ] = ∑ i ∑ j ( a x i + b y j ) P ( X = x i , Y = y j ) = a ∑ i x i P ( X = x i ) + b ∑ j y j P ( Y = y j ) = a E [ X ] + b E [ Y ] = a \sum_i x_i P(X = x_i) + b \sum_j y_j P(Y = y_j) = aE[X] + bE[Y] = a ∑ i x i P ( X = x i ) + b ∑ j y j P ( Y = y j ) = a E [ X ] + b E [ Y ]
Worked example Discrete Example: Dice RollRoll a fair six-sided die. X ∈ { 1 , 2 , 3 , 4 , 5 , 6 } X \in \{1, 2, 3, 4, 5, 6\} X ∈ { 1 , 2 , 3 , 4 , 5 , 6 } P = 1 / 6 P = 1/6 P = 1/6
E [ X ] = 1 ⋅ 1 6 + 2 ⋅ 1 6 + ⋯ + 6 ⋅ 1 6 = 21 6 = 3.5 E[X] = 1 \cdot \frac{1}{6} + 2 \cdot \frac{1}{6} + \cdots + 6 \cdot \frac{1}{6} = \frac{21}{6} = 3.5 E [ X ] = 1 ⋅ 6 1 + 2 ⋅ 6 1 + ⋯ + 6 ⋅ 6 1 = 6 21 = 3.5
Why this step? Each outcome contributes its value weighted by how likely it is. The sum of 1 + 2 + ⋯ + 6 = 21 1+2+\cdots+6 = 21 1 + 2 + ⋯ + 6 = 21
Worked example Continuous Example: Uniform DistributionX ∼ Uniform ( 0 , 1 ) X \sim \text{Uniform}(0, 1) X ∼ Uniform ( 0 , 1 ) f ( x ) = 1 f(x) = 1 f ( x ) = 1 x ∈ [ 0 , 1 ] x \in [0,1] x ∈ [ 0 , 1 ]
E [ X ] = ∫ 0 1 x ⋅ 1 d x = [ x 2 2 ] 0 1 = 1 2 E[X] = \int_0^1 x \cdot 1 \, dx = \left[ \frac{x^2}{2} \right]_0^1 = \frac{1}{2} E [ X ] = ∫ 0 1 x ⋅ 1 d x = [ 2 x 2 ] 0 1 = 2 1
Why 0.5? The distribution is symmetric around 0.5, so the balance point is exactly in the middle.
Intuition Measuring SpreadVariance measures how far values typically are from the mean. It's the average squared distance from E [ X ] E[X] E [ X ]
Definition Mathematical DefinitionVar ( X ) = E [ ( X − μ ) 2 ] = E [ X 2 ] − ( E [ X ] ) 2 \text{Var}(X) = E[(X - \mu)^2] = E[X^2] - (E[X])^2 Var ( X ) = E [( X − μ ) 2 ] = E [ X 2 ] − ( E [ X ] ) 2 μ = E [ X ] \mu = E[X] μ = E [ X ] Var ( X ) \text{Var}(X) Var ( X ) σ 2 \sigma^2 σ 2 V [ X ] V[X] V [ X ]
Form 1 (definition): Average squared deviation from mean.
Var ( X ) = E [ ( X − μ ) 2 ] \text{Var}(X) = E[(X - \mu)^2] Var ( X ) = E [( X − μ ) 2 ]
Form 2 (computational): Expand the square.
Var ( X ) = E [ X 2 − 2 μ X + μ 2 ] \text{Var}(X) = E[X^2 - 2\mu X + \mu^2] Var ( X ) = E [ X 2 − 2 μ X + μ 2 ] = E [ X 2 ] − 2 μ E [ X ] + μ 2 = E[X^2] - 2\mu E[X] + \mu^2 = E [ X 2 ] − 2 μ E [ X ] + μ 2
Why this step? Use linearity of expectation. E [ X ] = μ E[X] = \mu E [ X ] = μ μ 2 \mu^2 μ 2
= E [ X 2 ] − 2 μ 2 + μ 2 = E [ X 2 ] − μ 2 = E[X^2] - 2\mu^2 + \mu^2 = E[X^2] - \mu^2 = E [ X 2 ] − 2 μ 2 + μ 2 = E [ X 2 ] − μ 2
Why Form 2? Easier to compute—calculate E [ X ] E[X] E [ X ] E [ X 2 ] E[X^2] E [ X 2 ]
Why a 2 a^2 a 2
Var ( a X ) = E [ ( a X ) 2 ] − ( E [ a X ] ) 2 = a 2 E [ X 2 ] − a 2 ( E [ X ] ) 2 = a 2 Var ( X ) \text{Var}(aX) = E[(aX)^2] - (E[aX])^2 = a^2 E[X^2] - a^2 (E[X])^2 = a^2 \text{Var}(X) Var ( a X ) = E [( a X ) 2 ] − ( E [ a X ] ) 2 = a 2 E [ X 2 ] − a 2 ( E [ X ] ) 2 = a 2 Var ( X )
Why shift invariant? Adding a constant moves the distribution but doesn't change spread.
Var ( X + b ) = E [ ( X + b − E [ X + b ] ) 2 ] = E [ ( X − E [ X ] ) 2 ] = Var ( X ) \text{Var}(X + b) = E[(X + b - E[X+b])^2] = E[(X - E[X])^2] = \text{Var}(X) Var ( X + b ) = E [( X + b − E [ X + b ] ) 2 ] = E [( X − E [ X ] ) 2 ] = Var ( X )
Worked example Variance of a Dice RollFrom before, E [ X ] = 3.5 E[X] = 3.5 E [ X ] = 3.5
First, find E [ X 2 ] E[X^2] E [ X 2 ] E [ X 2 ] = 1 2 ⋅ 1 6 + 2 2 ⋅ 1 6 + ⋯ + 6 2 ⋅ 1 6 = 1 + 4 + 9 + 16 + 25 + 36 6 = 91 6 E[X^2] = 1^2 \cdot \frac{1}{6} + 2^2 \cdot \frac{1}{6} + \cdots + 6^2 \cdot \frac{1}{6} = \frac{1 + 4 + 9 + 16 + 25 + 36}{6} = \frac{91}{6} E [ X 2 ] = 1 2 ⋅ 6 1 + 2 2 ⋅ 6 1 + ⋯ + 6 2 ⋅ 6 1 = 6 1 + 4 + 9 + 16 + 25 + 36 = 6 91
Why sum squares? We need the expected value of the squared variable.
Now:
Var ( X ) = 91 6 − ( 3.5 ) 2 = 91 6 − 49 4 = 182 − 147 12 = 35 12 ≈ 2.92 \text{Var}(X) = \frac{91}{6} - (3.5)^2 = \frac{91}{6} - \frac{49}{4} = \frac{182 - 147}{12} = \frac{35}{12} \approx 2.92 Var ( X ) = 6 91 − ( 3.5 ) 2 = 6 91 − 4 49 = 12 182 − 147 = 12 35 ≈ 2.92
Worked example Variance of Uniform(0, 1)E [ X ] = 0.5 E[X] = 0.5 E [ X ] = 0.5
Find E [ X 2 ] E[X^2] E [ X 2 ] E [ X 2 ] = ∫ 0 1 x 2 ⋅ 1 d x = [ x 3 3 ] 0 1 = 1 3 E[X^2] = \int_0^1 x^2 \cdot 1 \, dx = \left[ \frac{x^3}{3} \right]_0^1 = \frac{1}{3} E [ X 2 ] = ∫ 0 1 x 2 ⋅ 1 d x = [ 3 x 3 ] 0 1 = 3 1
Why x 3 / 3 x^3/3 x 3 /3 Antiderivative of x 2 x^2 x 2
Var ( X ) = 1 3 − ( 1 2 ) 2 = 1 3 − 1 4 = 1 12 ≈ 0.083 \text{Var}(X) = \frac{1}{3} - \left(\frac{1}{2}\right)^2 = \frac{1}{3} - \frac{1}{4} = \frac{1}{12} \approx 0.083 Var ( X ) = 3 1 − ( 2 1 ) 2 = 3 1 − 4 1 = 12 1 ≈ 0.083
Intuition Variance in Original UnitsVariance has units of [ data ] 2 [\text{data}]^2 [ data ] 2 Standard deviation is the square root of variance, giving us spread in the same units as the original data. It's more interpretable.
Why take the square root? To return to the original scale. If you measure heights in cm, variance is in cm², but SD is in cm.
Worked example SD of Dice RollFrom variance ≈ 2.92 \approx 2.92 ≈ 2.92 σ = 2.92 ≈ 1.71 \sigma = \sqrt{2.92} \approx 1.71 σ = 2.92 ≈ 1.71
Interpretation : On average, a dice roll is about 1.71 away from the mean of 3.5.
Worked example SD of Uniform(0, 1)σ = 1 / 12 ≈ 0.289 \sigma = \sqrt{1/12} \approx 0.289 σ = 1/12 ≈ 0.289
Interpretation : Values typically deviate 0.29 from the center (0.5).
Common mistake Mistake 1: Thinking Variance = Average DistanceWrong : Var ( X ) = E [ ∣ X − μ ∣ ] \text{Var}(X) = E[|X - \mu|] Var ( X ) = E [ ∣ X − μ ∣ ] Steel-man : This feels right because variance measures "typical distance from mean." Why doesn't it work?
Fix : Absolute values are hard to work with mathematically (not differentiable at zero). Squaring gives us nice algebraic properties (linearity, the E [ X 2 ] − μ 2 E[X^2] - \mu^2 E [ X 2 ] − μ 2
The absolute deviation measure is called Mean Absolute Deviation (MAD) , used in robust statistics but not standard.
Common mistake Mistake 2: Var(X + Y) = Var(X) + Var(Y) AlwaysWrong : This only holds if X X X Y Y Y independent (or uncorrelated).
Steel-man : Intuitively, if both variables are "spread out," the sum should be more spread out by the sum of spreads.
Fix : The general formula is:
Var ( X + Y ) = Var ( X ) + Var ( Y ) + 2 Cov ( X , Y ) \text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X, Y) Var ( X + Y ) = Var ( X ) + Var ( Y ) + 2 Cov ( X , Y ) X X X Y Y Y more than the sum. If negatively correlated, less .
Common mistake Mistake 3: SD(X + Y) = SD(X) + SD(Y)Wrong : Standard deviations don't add linearly.
Steel-man : Since SD measures spread, adding two variables should add their spreads.
Fix : For independent variables:
SD ( X + Y ) = Var ( X ) + Var ( Y ) \text{SD}(X + Y) = \sqrt{\text{Var}(X) + \text{Var}(Y)} SD ( X + Y ) = Var ( X ) + Var ( Y )
Variances add, then you take the square root. Example: SD ( X ) = 3 \text{SD}(X) = 3 SD ( X ) = 3 SD ( Y ) = 4 \text{SD}(Y) = 4 SD ( Y ) = 4 SD ( X + Y ) = 9 + 16 = 5 \text{SD}(X+Y) = \sqrt{9 + 16} = 5 SD ( X + Y ) = 9 + 16 = 5
Model Evaluation : Variance measures prediction uncertainty. High variance = model is inconsistent.Bias-Variance Tradeoff : Balancing systematic error (bias) vs. prediction spread (variance).Gradient Descent : Variance of gradient estimates affects learning stability (see: SGD, mini-batch size).Feature Scaling : Standardization uses mean and SD to normalize features: z = x − μ σ z = \frac{x - \mu}{\sigma} z = σ x − μ Probabilistic Models : Gaussian distributions are parameterized by μ \mu μ σ 2 \sigma^2 σ 2 Loss Functions : MSE (Mean Squared Error) is essentially variance of prediction errors.
Recall Explain to a 12-Year-OldImagine you and your friends are throwing darts at a board.
Expectation is where you aim —the bullseye. If everyone throws 100 times, the expectation is the average spot where all the darts landed.
Variance is how mesy your throws are. If all darts cluster tightly near the bullseye, variance is small. If they're scattered all over, variance is big. We measure it by looking at how far each dart is from the average spot, squaring those distances (so left and right don't cancel), and averaging them.
Standard deviation is the same as variance, but we take the square root at the end so it's in the same units—like "on average, darts land 5 cm away from the bullseye" instead of "25 cm²."
In AI, we use these to know if a model is consistent (low variance) or all over the place (high variance).
E xpectation: E xpect where the center is.
V ariance: V ery squared distances (emphasizes outliers).
S tandard D eviation: S quare root D escent back to original units.
Formula chain: μ → σ 2 → σ \mu \to \sigma^2 \to \sigma μ → σ 2 → σ
#flashcards/ai-ml
What is the expectation of a discrete random variable? :: E [ X ] = ∑ i x i ⋅ P ( X = x i ) E[X] = \sum_i x_i \cdot P(X = x_i) E [ X ] = ∑ i x i ⋅ P ( X = x i )
What does variance measure? The average squared distance of values from the mean; quantifies spread or dispersion.
What is the computational formula for variance? Var ( X ) = E [ X 2 ] − ( E [ X ] ) 2 \text{Var}(X) = E[X^2] - (E[X])^2 Var ( X ) = E [ X 2 ] − ( E [ X ] ) 2 How does variance scale with a constant multiplier? Var ( a X ) = a 2 Var ( X ) \text{Var}(aX) = a^2 \text{Var}(X) Var ( a X ) = a 2 Var ( X ) (scaling by
a a a scales variance by
a 2 a^2 a 2 ).
What is standard deviation? The square root of variance,
σ = Var ( X ) \sigma = \sqrt{\text{Var}(X)} σ = Var ( X ) , giving spread in original units.
For independent RVs, how do variances combine? Var ( X + Y ) = Var ( X ) + Var ( Y ) \text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) Var ( X + Y ) = Var ( X ) + Var ( Y ) (variances add).
Why do we square deviations in variance? To prevent positive and negative deviations from canceling, and to give algebraic properties like
E [ X 2 ] − μ 2 E[X^2] - \mu^2 E [ X 2 ] − μ 2 .
Does expectation obey linearity? Yes,
E [ a X + b Y ] = a E [ X ] + b E [ Y ] E[aX + bY] = aE[X] + bE[Y] E [ a X + bY ] = a E [ X ] + b E [ Y ] for any constants
a , b a, b a , b .
What is the variance of a constant? Zero,
Var ( c ) = 0 \text{Var}(c) = 0 Var ( c ) = 0 , because a constant has no spread.
If E [ X ] = 5 E[X] = 5 E [ X ] = 5 E [ X 2 ] = 30 E[X^2] = 30 E [ X 2 ] = 30 Var ( X ) \text{Var}(X) Var ( X ) Var ( X ) = 30 − 5 2 = 30 − 25 = 5 \text{Var}(X) = 30 - 5^2 = 30 - 25 = 5 Var ( X ) = 30 − 5 2 = 30 − 25 = 5 .
Continuous integral formula
Intuition Hinglish mein samjhoDekho, expectation matlab long-run average hai. Agar tum ek dice 1000 baar roll karo, toh average value 3.5 ayega—yeh expectation hai. Formula simple hai: har possible value ko uski probability se multiply karo aur sab add kar do. Continuous distributions ke liye integral use hota hai, but logic same hai.
Variance bata hai ki data kitna spread out hai mean se. Agar sab values mean ke paas hain, variance chhota. Agar values scattered hain, variance bada. Formula: har deviation ko square karo (taki positive-negative cancel na ho), phir average nikalo. Computation ke liye shortcut: E [ X 2 ] − ( E [ X ] ) 2 E[X^2] - (E[X])^2 E [ X 2 ] − ( E [ X ] ) 2
Standard deviation variance ka square root hai—yeh original units mein spread bata hai. Agar height cm mein hai, variance cm² mein hoga but SD phir se cm mein. ML mein yeh feature scaling (standardization), model consistency (low variance = stable predictions), aur bias-variance tradeoff mein kaam ata hai. Samjho: expectation = center, variance = spread, SD = spread in original units. Teen concepts, ek chain: μ → σ 2 → σ \mu \to \sigma^2 \to \sigma μ → σ 2 → σ
