Foundations — Entropy and KL divergence

This page builds every piece of notation the parent note topic note leans on, starting from a smart 12-year-old who has never seen a $\sum$ sign. Read it top to bottom; each symbol is earned before it is used.

1. A probability $p(x)$ — a slice of certainty

Before any formula, we need the raw material: probability.

The picture. Imagine a bar of total length $1$. You cut it into pieces, one piece per possible outcome. The width of a piece is its probability. All widths together must exactly fill the bar — nothing missing, nothing overflowing.

Why the topic needs it. Entropy and KL divergence are averages weighted by these widths. A wide (likely) outcome pulls the average toward its value; a thin (rare) outcome barely nudges it. Without probabilities there is nothing to weigh.

2. The random variable $X$ and its alphabet $\mathcal{X}$

Read $x\in \mathcal{X}$ as "$x$ is one item chosen from the list $\mathcal{X}$." The symbol $\in$ just means "belongs to."

The picture. $\mathcal{X}$ is the set of labels written under each slice of the certainty bar in §1. If the bar has three slices, then $\mathcal{X}=\{\text{sun},\text{rain},\text{snow}\}$ and $|\mathcal{X}|=3$ — the vertical bars $|\cdots |$ mean "count how many items are inside."

Why the topic needs it. The maximum-entropy property $H(X)\le {\log }_2|\mathcal{X}|$ literally counts how many outcomes exist. That $|\mathcal{X}|$ is exactly this list-length.

3. The sum sign $\sum$ — "add up over every slice"

Every formula on the parent page begins with $\sum$. It is not scary.

For $\mathcal{X}=\{a,b,c\}$: ${\sum }_{x\in \mathcal{X}}f(x)=f(a)+f(b)+f(c)$

The picture. Go slice by slice along the certainty bar, drop each slice's contribution into a bucket, and read off the bucket at the end.

4. The logarithm $\log$ — turning "how many halvings?" into a number

This is the tool the whole topic is built on, so we install it carefully.

Examples you can check by hand:

• ${\log }_{2}1=0$ (already there, no doublings).
• ${\log }_{2}2=1$, ${\log }_{2}4=2$, ${\log }_{2}8=3$.
• ${\log }_2\frac{1}{2}=-1$ (one halving — negative because we shrank).

The additivity that earns the log. If event $x$ has probability $p$ and independent event $y$ has probability $q$, the pair happens with probability $p\cdot q$. Then $-{\log }_2(p\cdot q)=-{\log }_{2}p+(-{\log }_{2}q).$ Surprise of the pair = surprise of each, added. That single line is why $-{\log }_{2}p$ (called surprisal) is the right meter, and why $\log$ sits at the heart of entropy.

5. Surprisal $I(x)=-{\log }_{2}p(x)$ — the height of one slice's shock

The picture. Rare slice (thin, small $p$) → tall surprise spike. Certain slice ($p=1$) → flat zero. Plot surprisal against probability and you get a curve diving to $0$ at $p=1$ and shooting to $+\infty$ as $p\to 0$.

Why the topic needs it. Entropy is nothing but the average of these spike heights, and cross-entropy is the average of spikes computed with the wrong meter — so surprisal is the atom both are built from.

6. Expectation ${\mathbb{E}}_{x\sim p}[\cdot ]$ — the weighted average

The picture. Balance point of a see-saw: place a weight $p(x)$ at position $f(x)$ for every outcome; the expectation is where the plank balances. Wide slices sit heavier and drag the balance toward their value.

Why the topic needs it. With this one symbol the parent's two headline formulas collapse into plain English: $H(X)={\mathbb{E}}_{x\sim p}[I(x)]\text{(average surprise)},D_{KL}(p\Vert q)={\mathbb{E}}_{x\sim p}\left[\log \frac{p(x)}{q(x)}\right]\text{(average wasted surprise)}.$

7. Two distributions $p$ and $q$ — truth vs. your guess

The picture. Two certainty bars stacked. Where the $q$-slice is narrower than the $p$-slice, your model under-expects a common event — and you pay for it. KL divergence is the running tally of that payment.

8. The conventions that stop the formulas from exploding

The picture. As a slice's width shrinks toward zero, its surprisal spike grows tall, but the width multiplying it shrinks faster — the product (a thin, tall sliver of area) collapses to nothing. The see-saw feels no weight from a zero-width slice.

Prerequisite map

This map shows the dependency order used above: probabilities and logs are the roots; expectation fuses surprisal into entropy; two distributions plus the log-ratio give KL divergence; both feed the parent topic. Downstream, these same tools power Cross-Entropy Loss, Mutual Information, Information Gain, the Maximum Entropy Principle, Jensen-Shannon Divergence, F-divergences, and the Evidence Lower Bound (ELBO) used in Variational Autoencoders.

Equipment checklist

Cover the right side and test yourself. If you can answer all of these, you are ready for the parent note.