Foundations — Expectation, variance, and standard deviation

Before you can read the parent note, you need to own every squiggle it uses. Below is every symbol and idea, in build order — plain words first, then a picture, then why the topic can't live without it.

1. A random variable $X$

Plain words: roll a die — the outcome is a dot pattern, but $X$ turns that into a number $1,2,\dots ,6$. The capital $X$ is the machine; a lowercase $x$ (or $x_i$) is one specific number it can spit out.

The picture: think of a bag of possible values, each ball a number.

Why the topic needs it: expectation and variance are questions about $X$. Without a number-producing machine there is nothing to average or to measure the spread of. This is built in detail in 1.3.01-Randomvariables-and-probability-distributions.

2. Probability $P(X=x_i)$

Plain words: $P=0$ means "never," $P=1$ means "always," $P=\frac{1}{6}$ means "one time in six."

Look at the figure: each value on the horizontal axis gets a bar whose height is its probability. Two ironclad rules — every bar height is $\ge 0$, and all bar heights add up to exactly $1$ (something must happen).

Why the topic needs it: expectation weights each value by how likely it is. The weights ARE these probabilities.

3. The summation sign $\sum$

Plain words: ${\sum }_{i=1}^3{x}_i=x_1+x_2+x_3$. The big Greek $\Sigma$ (sigma) is just a compact "+".

The picture: a conveyor belt — each item $i$ rolls past, you drop its value into a running total.

Why the topic needs it: expectation for a discrete variable is "add up each value times its chance," which is exactly a $\sum$.

4. The weighted average — the heart of expectation

Look at the figure: it is a seesaw. Each value sits at its position on the plank; the size of the block at that position is its probability. The single point where the plank balances is the expectation $E[X]$. That is why people call it the "centre of mass."

Why this and not a plain average? Because outcomes are not equally likely in general. The probability is the correct weight — it is literally how often that value shows up in the long run.

5. The symbols $E[X]$, $\mu$, $⟨X⟩$

Why so many names? History. Just recognise them as identical so no note can surprise you.

6. Deviation $X-\mu$

Look at the figure: each value has an arrow back to the centre line $\mu$. Arrows pointing right are positive deviations (green), arrows pointing left are negative (red).

Why the topic needs it: variance is about how far values sit from the centre — the deviation is exactly that distance, and we must handle both signs, which motivates the next idea.

7. Squaring $(X-\mu {)}^2$ — why square, not absolute value

We need a way to make every deviation count as "distance," ignoring direction. Two candidates:

• Absolute value $|X-\mu |$ — throws away the minus sign.
• Square $(X-\mu {)}^2$ — a squared number is always $\ge 0$.

The topic squares. Why square and not absolute value? Two concrete reasons:

1. Squaring is smooth everywhere — you can do calculus on it. Absolute value has a sharp corner at zero (not differentiable), which breaks later machinery like gradient descent.
2. Squaring gives the clean algebra $E[X^2]-{\mu }^2$ that makes hand-computation easy.

8. Square root  ​ and standard deviation $\sigma$

Plain words: if $X$ is a height in cm, then $(X-\mu {)}^2$ is in cm² — an awkward unit. The square root drags it back to plain cm, so you can say "typically about $\sigma$ cm away from the mean."

Why the topic needs it: interpretability. Variance is the algebra-friendly number; standard deviation is the human-friendly number. Feature scaling (4.1.02-Feature-scaling-and-normalization) divides by $\sigma$ precisely because it lives in real units.

9. The integral $\int$ — expectation for continuous variables

The integral sign $\int$ is just the $\sum$ from before, taken to the limit of infinitely many infinitely thin slivers:

$E[X]={\int }_{-\infty }^{\infty }xf(x)dx$

Read the pieces: $x$ is the value, $f(x)dx$ is the tiny probability of that value's sliver (a weight), and $\int$ adds every "value $\times$ its tiny chance" — exactly the weighted average from §4, now with a curve instead of bars.

The figure below shows the term-by-term translation between the discrete sum and the continuous integral — the same weighted-average idea, once with bars and once with a smooth curve.

Why the topic needs it: real ML features (weights, pixel intensities, times) are continuous. You must recognise $\int$ as "the same weighted-average idea, now with a curve."

10. Constants $a,b,c$ and linearity

The picture: $aX$ stretches the seesaw plank; $X+b$ slides it sideways without stretching.

Why the topic needs it: the property list (linearity, scaling, shift-invariance) is written entirely in terms of $a,b,c$.

Prerequisite map

This map feeds directly into Expectation, variance, and standard deviation and onward to 1.3.09-Covariance-and-correlation, 1.3.12-Central-limit-theorem, and 3.2.04-Bias-variance-tradeoff.

Equipment checklist

Test yourself — cover the right side.