Entropy

The unifying principle

Start with a probability distribution $p_i$ over possible states $i$ . Define the Shannon entropy:

$H(p) = -\sum_i p_i \log p_i$

This single formula is the bridge. It is maximized when $p_i$ is uniform (all states equally likely → maximal ignorance) and zero when one state has $p_i = 1$ (certainty).

If all $W$ microstates are equally likely, $p_i = 1/W$ , and it collapses to:

$H = -\sum_{i=1}^{W} \frac{1}{W}\log\frac{1}{W} = \log W$

This is Boltzmann's formula (with a unit constant). So thermodynamic entropy is just the uniform-distribution special case of information entropy. The choice of logarithm base sets the units:

base 2 → bits (information)
base $e$ → nats (statistics/ML)
multiply by Boltzmann's $k_B$ → J/K (thermodynamics)

Everything below is this one expression wearing different clothes.

Form: $S = k_B \ln W$ (Boltzmann), or $S = -k_B \sum_i p_i \ln p_i$ (Gibbs). Why it's the same idea: $W$ is the number of microstates consistent with the macrostate — literally "the number of arrangements you can't distinguish." The Second Law ( $\Delta S \geq 0$ ) says isolated systems drift toward the macrostate with the most microstates, i.e. maximal ignorance. Example: Free expansion of an ideal gas into double the volume. Each molecule now has twice the accessible positions, so $W \to W \cdot 2^N$ and $\Delta S = k_B \ln(2^N) = N k_B \ln 2$ . For one mole, $\Delta S = R\ln 2 \approx 5.76\ \text{J/K}$ .

Chemistry (Reaction spontaneity)

Form: Gibbs free energy $\Delta G = \Delta H - T\Delta S$ . A reaction is spontaneous when $\Delta G < 0$ . Why it's the same idea: $\Delta S$ counts how the number of accessible molecular arrangements (translational, rotational, vibrational modes) changes across the reaction. Dissolving, melting, or producing more gas moles raises $W$ and thus $S$ . Example: $\text{NH}_4\text{NO}_3(s) \to \text{NH}_4^+(aq) + \text{NO}_3^-(aq)$ . This dissolution is endothermic ( $\Delta H > 0$ , the pack gets cold) yet spontaneous — because the ordered crystal disperses into many mobile solvated arrangements, so $\Delta S > 0$ and $T\Delta S$ wins.

AI / Machine Learning (Cross-entropy loss)

Form: For true distribution $p$ (one-hot labels) and model prediction $q$ : $H(p, q) = -\sum_i p_i \log q_i$ Related: KL divergence $D_{KL}(p\|q) = H(p,q) - H(p)$ measures how far $q$ is from $p$ . Why it's the same idea: Cross-entropy is the average surprise of the true labels under the model's beliefs. Minimizing it makes the model less surprised by reality. Since $H(p,q) \geq H(p)$ with equality iff $q = p$ , the loss floor is the data's intrinsic entropy. Example: A classifier predicts $q = [0.7, 0.2, 0.1]$ but the truth is class 1 ( $p = [1,0,0]$ ). Loss $= -\log 0.7 \approx 0.357$ nats. If it had predicted $0.99$ , loss $\approx 0.01$ — near-zero surprise.

Coding / CS (Data compression)

Form: Shannon's source coding theorem: the minimum average bits per symbol is $H(p)$ (in base 2). Why it's the same idea: You can't compress below the entropy — it's the irreducible information content. Frequent symbols get short codes; rare (surprising) symbols get long ones, matching $-\log p_i$ bits per symbol. Example: Text where 'e' appears with $p=0.5$ , others rare. Huffman/arithmetic coding assigns 'e' ~1 bit and rare letters many bits. A source with $H = 2.1$ bits/char cannot be losslessly stored below 2.1 bits/char on average — that's the entropy wall.

Why this bridge matters

What transfers:

Physics → ML: The maximum entropy principle — "assume the least, be maximally uncertain given constraints" — derives the Boltzmann/softmax distribution $p_i \propto e^{-E_i/T}$ . This is literally the softmax in neural nets, with logits as negative energies and a temperature $T$ that flattens or sharpens predictions.
Information theory → Chemistry: Seeing $S$ as "missing information" demystifies why endothermic reactions can be spontaneous: nature maximizes accessible arrangements, not minimizes energy.
ML → CS: Cross-entropy loss and compression are dual — a good language model is a good compressor (this is the basis of the "compression = intelligence" viewpoint; e.g. arithmetic coding driven by an LLM).

The unlocking intuition: Once you see that "disorder" (physics), "product dispersal" (chemistry), "model surprise" (ML), and "irreducible bits" (CS) are all $-\sum p_i \log p_i$ , temperature, softmax, spontaneity, and compression limits stop being separate facts. They're one theorem in four costumes.

Connections

01 Statistical Mechanics & Microstates
02 Second Law of Thermodynamics
03 Gibbs Free Energy & Spontaneity
04 Shannon Information Theory
05 Cross-Entropy & KL Divergence
06 Softmax & Maximum Entropy
07 Lossless Compression & Source Coding
08 Boltzmann Distribution

#bridge