Foundations — Direct Preference Optimization (DPO)

This page assumes you have seen none of the notation in the Direct Preference Optimization (DPO) parent note. We build every piece from the ground up: what it means in plain words, the picture it stands for, and why DPO can't work without it. Read top to bottom — each symbol is earned before it is used.

0. What is a language model, as a probability machine?

Before any Greek letters, fix the mental picture.

Look at the figure: the same prompt "The capital of France is" gives a tall bar to "Paris" (high probability) and a tiny bar to "banana" (low probability). A language model is nothing more than this bar-height assignment, done over every possible continuation.

We need this picture because everything in DPO is about nudging these bar heights — up for good answers, down for bad ones.

1. The symbols for prompt and responses: $x,y_w,y_l$

The picture: for one prompt $x$, a human was shown two answers side by side and pointed at one. That pointing is written $y_w\succ y_l$, read "$y_w$ is preferred over $y_l$."

DPO needs pairs, not single answers, because its whole trick works by comparing two responses to the same $x$ (you will see why in Section 7).

2. Conditional probability notation: $\pi (y\mid x)$

The vertical bar $\mid$ is the single most-used symbol in the parent note. It looks scary; it means one plain thing.

The picture: cover up the right side of the bar with your hand — you've fixed the prompt to "The capital of France is". Now the left side ($y$) ranges over all possible answers, and $\pi$ gives each a bar height (Section 0's figure). Change $x$ and you get a different set of bar heights.

We use the Greek letter $\pi$ (pi) purely by tradition — in this field $\pi$ stands for policy, a model that produces actions (here, text). It has nothing to do with $3.14159$.

3. The two models: ${\pi }_{\theta }$ and ${\pi }_{\text{ref}}$

DPO always juggles two copies of the model.

The picture: two identical twins at the start. One (the reference) is turned to stone — it will never move. The other (the policy) is free to shift its bar heights during training. DPO always asks "how far has the moving twin drifted from the frozen twin?"

Why keep a frozen copy at all? Because if we only pushed absolute probabilities up, the model would collapse onto a few generic high-probability sentences (word soup that's technically likely). Measuring relative to ${\pi }_{\text{ref}}$ keeps the model's overall style sane. This is also the reason $\theta$ appears as a subscript only on the trainable one.

4. The log-ratio: $\log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$

This fraction-inside-a-log is the heart of DPO. Build it in two moves.

Move A — the ratio. $\frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$ compares the two twins on the same response $y$:

• ratio $>1$: the trained model made $y$ more likely than the frozen one did.
• ratio $=1$: no change.
• ratio $<1$: the trained model made $y$ less likely.

So the ratio is a pure "how much did we move this answer?" dial, immune to how likely the answer already was.

Move B — the log. We take the [natural] logarithm of that ratio.

Look at the figure: the horizontal axis is the ratio, the curve is $\log (\text{ratio})$. It crosses zero exactly where the ratio is $1$ (the frozen twin), dives to $-\infty$ as the answer is suppressed, rises slowly as it's boosted.

5. Turning the log-ratio into a reward: $\beta$ and ${\hat{r}}_{\theta }$

The picture: the log-ratio measures drift; $\beta$ is the exchange rate turning "amount of drift" into "amount of reward." Doubling $\beta$ doubles the reward you get for the same drift — so the model can reach a target reward with less drift. That is why large $\beta$ = tighter leash (a common trap the parent note steel-mans).

Why call it "implicit"? Because classic RLHF - Reinforcement Learning from Human Feedback trains a whole separate network for reward. DPO discovers that the policy itself, compared to its frozen twin, already encodes a reward — no extra network needed. See Reward Modeling for the thing DPO replaces.

6. The sigmoid $\sigma$: squashing a score into a probability

We now have, per response, a reward number that can be any size (positive or negative). To talk about "probability that the winner beats the loser," we must squash a number into $[0,1]$. That's the sigmoid's job.

Look at the S-shaped curve:

• $z=0\Rightarrow \sigma =0.5$ (a coin flip — no evidence either way).
• large positive $z\Rightarrow \sigma \to 1$ ("almost certainly the winner").
• large negative $z\Rightarrow \sigma \to 0$ ("almost certainly not").

Why this function and not, say, clipping at 0 and 1? Because $\sigma$ is smooth everywhere, so we can take its derivative and do gradient descent, and it never actually reaches $0$ or $1$, so the loss (Section 8) never blows up to infinity. Full detail lives in Sigmoid and Logistic Loss.

7. The Bradley–Terry model: comparing two rewards

The picture: line up the two rewards. Only the gap between them matters — if the winner's reward is well above the loser's, the difference is a big positive number and $\sigma$ says "≈ 1, yes the human preferred correctly."

8. The loss $\mathcal{L}$ and the expectation $\mathbb{E}$

Reading it right to left:

1. ${\hat{r}}_w-{\hat{r}}_l$ = "by how much does the model reward the winner over the loser?" (Section 5, 7).
2. $\sigma (\cdot )$ = turn that gap into "probability the model agrees with the human" (Section 6).
3. $\log$ = a well-behaved way to score that probability (big penalty as it $\to 0$).
4. minus sign + $\mathbb{E}$ = we minimise the average, and minimising $-\log (\text{probability})$ is the same as maximising the probability of getting every human preference right.

The picture: it's exactly a yes/no classifier — "did the model rank this pair the way the human did?" That's why the parent note calls DPO "alignment as a classification loss," sidestepping the RL machinery of PPO - Proximal Policy Optimization.

Prerequisite map

Each arrow is "you must understand the left box before the right one makes sense." Notice the sigmoid ($G$) joins in only at the Bradley–Terry step — it is the translator, not a foundation stone.

Equipment checklist

Say each answer aloud before revealing.

When all feel automatic, open Direct Preference Optimization (DPO) and the derivation will read like plain English.