Expectation, variance, and standard deviation: Exercises

The three tools we lean on all page long:

Above: think of expectation as the point where the "mass" of probability balances (magenta pivot), and variance/standard deviation as how wide the mass is smeared (violet band). Keep this picture in mind for every problem.

Level 1 — Recognition

These test whether you can read a distribution and plug into the definition.

Problem 1.1

A coin is flipped once. Let $X = 1$ if heads, $X = 0$ if tails, with $P (heads) = 0.5$ . Find $E [X]$ .

Recall Solution 1.1

What we do: apply $E [X] = \sum x_{i} P (X = x_{i})$ directly. $E [X] = 1 \cdot 0.5 + 0 \cdot 0.5 = 0.5$ Why: each value is weighted by how likely it is. Heads contributes $1 \times 0.5$ , tails contributes nothing because its value is $0$ . Answer: $E [X] = 0.5$ .

Problem 1.2

A random variable $Y$ takes values ${- 2, 0, 3}$ with probabilities ${0.25, 0.5, 0.25}$ . Find $E [Y]$ .

Recall Solution 1.2

$E [Y] = (- 2) (0.25) + (0) (0.5) + (3) (0.25) = - 0.5 + 0 + 0.75 = 0.25$ Why the negative term: a negative value pulls the balance point left, exactly as a mass placed on the left of a seesaw does. Answer: $E [Y] = 0.25$ .

Problem 1.3

State which is measured in the same units as the data: variance or standard deviation? Why?

Recall Solution 1.3

Standard deviation. Variance is an average of squared deviations, so it carries units of $[data]^{2}$ (e.g. $cm^{2}$ ). Taking the square root, $σ = Var (X)$ , returns to $cm$ — the original scale, which is why it is more interpretable. This is exactly why feature scaling uses $σ$ , not $σ^{2}$ , in $z = \frac{x - μ}{σ}$ .

Level 2 — Application

Now compute variance and SD, and use the scaling/shift rules.

Problem 2.1

For the coin $X$ of Problem 1.1 ( $E [X] = 0.5$ ), find $Var (X)$ and $σ$ .

Recall Solution 2.1

Step 1 — compute $E [X^{2}]$ . Because $X \in {0, 1}$ , note $X^{2} = X$ , so $E [X^{2}] = E [X] = 0.5$ . Step 2 — apply Form 2: $Var (X) = E [X^{2}] - (E [X])^{2} = 0.5 - 0.25 = 0.25$ . Step 3 — SD: $σ = 0.25 = 0.5$ . Why Form 2: computing $E [X^{2}]$ then subtracting $μ^{2}$ avoids re-summing $(x - μ)^{2}$ term by term. Answer: $Var (X) = 0.25$ , $σ = 0.5$ .

Problem 2.2

Let $Var (X) = 9$ . A new variable is $W = 3 X + 7$ . Find $Var (W)$ and $SD (W)$ .

Recall Solution 2.2

Scaling rule: $Var (a X + b) = a^{2} Var (X)$ , so the $+ 7$ shift does nothing to spread and $a = 3$ multiplies variance by $a^{2} = 9$ . $Var (W) = 3^{2} \cdot 9 = 81$ $SD (W) = 81 = 9 = ∣3∣ \cdot 9 = ∣3∣ \cdot 3$ Why $+ 7$ vanishes: adding a constant slides the whole distribution sideways — the shape and width are untouched, so spread cannot change. Answer: $Var (W) = 81$ , $SD (W) = 9$ .

Problem 2.3

$X \sim Uniform (0, 4)$ , so $f (x) = \frac{1}{4}$ on $[0, 4]$ . Find $E [X]$ and $Var (X)$ .

Recall Solution 2.3

Mean by integral: $E [X] = \int_{0}^{4} x \cdot \frac{1}{4} d x = \frac{1}{4} [\frac{x ^{2}}{2}]_{0}^{4} = \frac{1}{4} \cdot 8 = 2$ Second moment: $E [X^{2}] = \int_{0}^{4} x^{2} \cdot \frac{1}{4} d x = \frac{1}{4} [\frac{x ^{3}}{3}]_{0}^{4} = \frac{1}{4} \cdot \frac{64}{3} = \frac{16}{3}$ Variance: $Var (X) = \frac{16}{3} - 2^{2} = \frac{16}{3} - 4 = \frac{4}{3} \approx 1.333$ . Sanity check: general Uniform $(a, b)$ variance is $\frac{( b - a ) ^{2}}{12} = \frac{16}{12} = \frac{4}{3}$ . ✓ Answer: $E [X] = 2$ , $Var (X) = \frac{4}{3}$ .

Level 3 — Analysis

Combine variables and reason about which rule applies.

Problem 3.1

$X$ and $Y$ are independent dice-like variables with $Var (X) = \frac{35}{12}$ and $Var (Y) = \frac{35}{12}$ . Find $Var (X + Y)$ and $SD (X + Y)$ .

Recall Solution 3.1

Independence lets variances add: $Var (X + Y) = Var (X) + Var (Y) = \frac{35}{12} + \frac{35}{12} = \frac{70}{12} = \frac{35}{6} \approx 5.833$ . SD: $SD (X + Y) = 35/6 \approx 2.415$ . Why not add SDs: $35/12 + 35/12 \approx 1.708 + 1.708 = 3.416 \neq = 2.415$ . Variances add; SDs do not. Answer: $Var (X + Y) = \frac{35}{6}$ , $SD (X + Y) \approx 2.415$ .

Problem 3.2

$X, Y$ are not independent: $Var (X) = 4$ , $Var (Y) = 9$ , $Cov (X, Y) = - 3$ . Find $Var (X + Y)$ .

Recall Solution 3.2

Use the general formula (see covariance): $Var (X + Y) = Var (X) + Var (Y) + 2 Cov (X, Y) = 4 + 9 + 2 (- 3) = 7$ Why the sum shrank: negative covariance means when $X$ is high, $Y$ tends low — they partly cancel, so the combined spread is smaller than $4 + 9 = 13$ . Answer: $Var (X + Y) = 7$ .

Problem 3.3

A gradient estimate from one sample has variance $σ^{2} = 0.36$ . A mini-batch averages $B = 9$ independent estimates: $\overset{g}{ˉ} = \frac{1}{9} \sum_{i = 1}^{9} g_{i}$ . Find $Var (\overset{g}{ˉ})$ and $SD (\overset{g}{ˉ})$ .

Recall Solution 3.3

Step 1 — pull out the $\frac{1}{9}$ with the scaling rule ( $a = \frac{1}{9}$ , so $a^{2} = \frac{1}{81}$ ): $Var (\overset{g}{ˉ}) = \frac{1}{81} Var (\sum_{i = 1}^{9} g_{i}) .$ Step 2 — independent terms, so variances add: $Var (\sum g_{i}) = 9 \cdot 0.36 = 3.24$ . Step 3: $Var (\overset{g}{ˉ}) = \frac{3.24}{81} = 0.04$ , and $SD (\overset{g}{ˉ}) = 0.04 = 0.2$ . Insight: averaging $B$ independent estimates divides variance by $B$ (here $0.36/9 = 0.04$ ), so SD drops by $B$ . This is exactly why larger mini-batches give steadier updates in gradient descent variants. Answer: $Var (\overset{g}{ˉ}) = 0.04$ , $SD (\overset{g}{ˉ}) = 0.2$ .

Level 4 — Synthesis

Chain several tools inside one problem.

Problem 4.1

A feature has $μ = 50$ , $σ = 10$ . You standardise it: $Z = \frac{X - μ}{σ}$ . Prove $E [Z] = 0$ and $Var (Z) = 1$ .

Recall Solution 4.1

Write $Z = \frac{1}{σ} X - \frac{μ}{σ}$ , i.e. $Z = a X + b$ with $a = \frac{1}{σ}$ , $b = - \frac{μ}{σ}$ . Mean (linearity): $E [Z] = \frac{1}{σ} E [X] - \frac{μ}{σ} = \frac{μ}{σ} - \frac{μ}{σ} = 0.$ Variance (scaling, shift-invariant): the $b$ shift does nothing, and $a^{2} = \frac{1}{σ ^{2}}$ : $Var (Z) = \frac{1}{σ ^{2}} Var (X) = \frac{σ ^{2}}{σ ^{2}} = 1.$ Why this matters: standardisation always yields mean $0$ , SD $1$ , regardless of the original $μ, σ$ — the backbone of normalization. Plugging in the numbers ( $μ = 50, σ = 10$ ) is not even needed; the result is universal. Answer: $E [Z] = 0$ , $Var (Z) = 1$ .

Problem 4.2

Let $X$ be a fair die ( $E [X] = 3.5$ , $Var (X) = \frac{35}{12}$ ). Define $T = 2 X - 1$ . Find $E [T]$ , $Var (T)$ , and $SD (T)$ .

Recall Solution 4.2

Mean: $E [T] = 2 E [X] - 1 = 2 (3.5) - 1 = 6$ . Variance: shift $- 1$ irrelevant, factor $2$ squares: $Var (T) = 2^{2} \cdot \frac{35}{12} = \frac{140}{12} = \frac{35}{3} \approx 11.667$ . SD: $SD (T) = 35/3 \approx 3.416$ . (Equivalently $∣2∣ \cdot 35/12 = 2 (1.708) = 3.416$ .) Answer: $E [T] = 6$ , $Var (T) = \frac{35}{3}$ , $SD (T) \approx 3.416$ .

Level 5 — Mastery

Prove a general result and design with it.

Problem 5.1

Prove the computational identity $Var (X) = E [X^{2}] - (E [X])^{2}$ from the definition $Var (X) = E [(X - μ)^{2}]$ , where $μ = E [X]$ . State each rule you use.

Recall Solution 5.1

Step 1 — expand the square (algebra): $(X - μ)^{2} = X^{2} - 2 μ X + μ^{2} .$ Step 2 — take expectation, use linearity ( $E [a U + bV] = a E [U] + b E [V]$ ) and that $μ$ is a constant: $E [(X - μ)^{2}] = E [X^{2}] - 2 μ E [X] + μ^{2} .$ Step 3 — substitute $E [X] = μ$ : $= E [X^{2}] - 2 μ \cdot μ + μ^{2} = E [X^{2}] - 2 μ^{2} + μ^{2} = E [X^{2}] - μ^{2} .$ Step 4 — rewrite $μ^{2} = (E [X])^{2}$ : $Var (X) = E [X^{2}] - (E [X])^{2} . ■$ Why it's useful: you compute $E [X]$ and $E [X^{2}]$ once each and subtract — far less work than summing $(x_{i} - μ)^{2}$ over every outcome.

Problem 5.2

A Bernoulli variable has $X = 1$ with probability $p$ and $X = 0$ with probability $1 - p$ . Show $E [X] = p$ and $Var (X) = p (1 - p)$ . Then find which $p$ maximises the variance.

Recall Solution 5.2

Mean: $E [X] = 1 \cdot p + 0 \cdot (1 - p) = p$ . Second moment: since $X^{2} = X$ (values are $0$ and $1$ ), $E [X^{2}] = E [X] = p$ . Variance: $Var (X) = E [X^{2}] - (E [X])^{2} = p - p^{2} = p (1 - p)$ . Maximising: treat $v (p) = p (1 - p) = p - p^{2}$ . Its vertex (a downward parabola) is at $p = \frac{1}{2}$ , giving $v = \frac{1}{4}$ . So variance is largest, $0.25$ , when the outcome is maximally uncertain (a fair coin), and it is $0$ at $p = 0$ or $p = 1$ (a certain outcome has no spread). Answer: $E [X] = p$ , $Var (X) = p (1 - p)$ , maximised at $p = 0.5$ with variance $0.25$ .

Recall One-line self-check before you leave

Expectation is linear (shifts move it) ::: $E [a X + b] = a E [X] + b$ Variance kills shifts and squares scales ::: $Var (a X + b) = a^{2} Var (X)$ Independent sum ::: variances add, then square-root for SD Correlated sum ::: add $2 Cov (X, Y)$ as well