Visual walkthrough — Direct Preference Optimization (DPO)

Step 0 — The three characters, drawn as dice

Before any equation, let's fix what the words mean using the simplest possible picture: a language model answering a prompt is just rolling a loaded die over possible responses.

Look at the figure: two dice for the same prompt "Write a haiku". Each face is one possible response; the height of the bar is how probable that response is. The reference die (teal) is our starting point; the policy die (orange) is what we sculpt. Training = reshaping the orange bars so good answers rise and bad answers fall — while not letting orange wander too far from teal.

Step 1 — What we are trying to maximize (the RLHF goal)

We want the die to roll high-scoring answers on average, but we don't want it to mutate into a weird die that games the judge. That tension is the whole objective:

Term by term:

• ${\max }_{{\pi }_{\theta }}$ — "choose the die-shape $\theta$ that makes the whole thing as big as possible".
• $\mathbb{E}[r]$ — the average reward, our carrot.
• $\mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ — the KL divergence (see KL Divergence), a number measuring how different the orange die is from the teal die. Zero if identical, growing as they diverge.
• $\beta$ — read "beta", a knob (a positive number) setting how hard we yank on that leash.

The figure shows a tug-of-war: the orange arrow (reward) pulls the die toward high-scoring faces; the teal spring (KL leash) pulls it back toward the reference shape. $\beta$ is the stiffness of the spring.

Step 2 — The best die has a known shape

Here is a fact from optimization (the same trick used across PPO - Proximal Policy Optimization and entropy-regularized RL): the objective in Step 1 has an exact best answer. The optimal die ${\pi }^{\ast }$ is:

Term by term:

• ${\pi }_{\text{ref}}(y\mid x)$ — start from the frozen teal die's probability.
• $\exp (\cdot )$ — the exponential function $e^{(\cdot )}$. Why this tool? Because it turns a score (which can be negative) into a positive multiplier, and it's the exact function that pops out when you solve the KL-regularized maximization. Big reward ⇒ multiplier much bigger than 1 ⇒ bar shoots up. Negative reward ⇒ multiplier below 1 ⇒ bar shrinks.
• $\frac{1}{\beta }$ — divides the reward. Small $\beta$ ⇒ huge tilt (reward dominates). Large $\beta$ ⇒ gentle tilt (stay near teal). This is the same leash from Step 1, now living in the exponent.
• $Z(x)$ — the partition function, defined below.

In the figure: teal bars get multiplied by an orange exponential curve (tall where reward is high), giving raw tilted bars; then $Z(x)$ squeezes them so they re-sum to 1. The red box around $Z(x)$ marks it as "the thing we can't afford".

Step 3 — Flip it: read the reward off the die

We have "reward ⇒ best die". We now run it backwards: given the die, what reward produced it? Why? Because our real goal is to train the die directly; if the reward hides inside the die's shape, maybe we never need a separate reward model.

Take the natural logarithm $\log$ of both sides. Why the log? Log turns products into sums and undoes the exponential — it's the exact inverse tool for an expression built from $\exp$ and multiplication. Applying it:

Now isolate $r$ (multiply through by $\beta$ and rearrange):

Term by term:

• $\log \frac{{\pi }^{\ast }}{{\pi }_{\text{ref}}}$ — the log-ratio. Positive if the trained die made $y$ more likely than the reference did; negative if less; zero if unchanged. This ratio is the heart of DPO.
• $\beta$ — converts that policy shift into reward units.
• $\beta \log Z(x)$ — the leftover villain. Crucially it depends only on $x$, not on $y$. Every response to the same prompt carries the same $Z(x)$ term. Remember this — it's the pin we pull in Step 5.

The figure reads left-to-right: the reward of a response equals $\beta \times$(its log-ratio, drawn as an up/down arrow between orange and teal bars) $+$ a constant grey slab labelled "same for all $y$".

Step 4 — How humans express preference (Bradley–Terry)

Humans don't hand us numeric rewards; they hand us comparisons: "answer $y_w$ is better than answer $y_l$". ($y_w$ = winner, $y_l$ = loser.) We need a rule linking two hidden rewards to the probability a human picks the winner.

Why sigmoid and not something else? Because we're converting a reward gap (any real number) into a probability (must live in 0–1), and the sigmoid is the natural, smooth, monotonic bridge — the bigger the reward margin, the more sure the human is to pick the winner.

The figure plots $\sigma (z)$ with three dots marked: at $z=0$ (fifty-fifty), at a positive margin (winner likely), at a negative margin (we're wrong — winner unlikely). The horizontal axis is the reward margin $r_w-r_l$.

Step 5 — The villain cancels (the punchline)

Now substitute the Step-3 reward into the Step-4 margin. Write $r_w=r(x,y_w)$ and $r_l=r(x,y_l)$:

Group the $Z$ terms:

The figure stacks two grey $Z(x)$ slabs (identical height) and shows them cancelling with a red "✕", leaving only the two log-ratio arrows behind.

Step 6 — Assemble the loss

Replace the ideal die ${\pi }^{\ast }$ with our actual trainable die ${\pi }_{\theta }$, feed the surviving margin into Bradley–Terry, and ask the model to make the observed preference likely. Maximizing that likelihood is the same as minimizing its negative $\log$:

Term by term:

• $-\log \sigma (\cdot )$ — small when the winner is favoured (inside $\to 1$, $\log 1=0$), large when we're wrong.
• ${\hat{r}}_w-{\hat{r}}_l$ — the margin: how much more our die lifts the winner than the loser (relative to reference).
• ${\mathbb{E}}_{(x,y_w,y_l)}$ — average this over every preference triple in the dataset.

The figure is the loss curve $-\log \sigma (\text{margin})$ vs the margin: flat-and-low on the right (winner already preferred, little to learn), steep-and-high on the left (model wrong, big gradient). The dot at margin $=0$ sits at height $\log 2\approx 0.693$.

Step 7 — Edge & degenerate cases (never get surprised)

The one-picture summary

One flow, left to right: RLHF objective (carrot + leash) → closed-form optimal die (with the red $Z$ villain) → flip to read reward off the die → Bradley–Terry difference cancels $Z$ → one sigmoid classification loss on the log-ratios. The whole three-model, RL-heavy pipeline compresses into a single supervised loss.

Quick recall