Direct Preference Optimization (DPO): Worked examples

Everything below uses two quantities from the parent note. Let us re-say them in words so no symbol is unearned:

We will keep one fact handy for every example, so define it once:

The scenario matrix

Every DPO pair, no matter how weird, lands in one of these cells. The examples that follow cover all of them.

#	Cell class	Concretely	Example
A	Winner ahead ( $m > 0$ )	model already prefers $y_{w}$	Ex 1
B	Winner behind ( $m < 0$ )	model prefers the loser — a mistake	Ex 2
C	Perfect tie / degenerate start ( $m = 0$ )	$π_{θ} = π_{ref}$ , or ratios equal	Ex 3
D	Limit $m \to + \infty$	loss floor, saturation, dead gradient	Ex 4
E	Limit $m \to - \infty$	loss blow-up, exploding gradient	Ex 4
F	Same $Δ ℓ$ , sweep $β$ (small→large)	how the leash bites	Ex 5
G	Gradient sign / weight check	which way does each response move?	Ex 6
H	Real-world word problem	chatbot politeness pair, whole batch	Ex 7
I	Exam twist: sign trap	negative log-ratios that still satisfy the model	Ex 8

Figure — Direct Preference Optimization (DPO)

The worked examples

Worked example Example 1 — Cell A: winner comfortably ahead

Given $β = 0.1$ , $ℓ_{w} = 2.0$ , $ℓ_{l} = - 1.0$ .

Forecast: margin is clearly positive — do you expect the loss above or below $0.693$ ? Above or below $0.5$ ?

Rewards: $\overset{r}{^}_{w} = 0.1 \cdot 2.0 = 0.2$ , $\overset{r}{^}_{l} = 0.1 \cdot (- 1.0) = - 0.1$ . Why this step? $β$ converts a policy shift (log-ratio) into a reward scale — that's the definition of the implicit reward.
Margin: $m = 0.2 - (- 0.1) = 0.3$ . Why this step? The margin is all the loss ever sees; individual $ℓ$ values only matter through their difference.
$σ (0.3) = \frac{1}{1 + e ^{- 0.3}} = 0.5744$ . Why this step? Convert the reward gap into the Bradley–Terry probability that the human's winner really wins.
$L = - lo g (0.5744) = 0.5544$ .

Verify: $m > 0$ so we predicted $L < lo g 2 = 0.693$ . Indeed $0.554 < 0.693$ . ✓ The model is already on the right side, so the loss is modest but nonzero — there is still learning to do.

Worked example Example 2 — Cell B: the model is currently WRONG

Given $β = 0.1$ , $ℓ_{w} = - 0.5$ , $ℓ_{l} = 1.5$ (the model has raised the loser and lowered the winner — bad!).

Forecast: Will the loss be above or below the coin-flip $0.693$ ?

Rewards: $\overset{r}{^}_{w} = 0.1 \cdot (- 0.5) = - 0.05$ , $\overset{r}{^}_{l} = 0.1 \cdot 1.5 = 0.15$ . Why this step? Same conversion, but now the loser earns more reward than the winner.
Margin: $m = - 0.05 - 0.15 = - 0.2$ . Negative. Why this step? Negative margin is the mathematical fingerprint of "model ranks the pair backwards."
$σ (- 0.2) = \frac{1}{1 + e ^{0.2}} = 0.4502$ .
$L = - lo g (0.4502) = 0.7981$ .

Verify: $m < 0 \Rightarrow L > lo g 2$ . Indeed $0.798 > 0.693$ . ✓ The parent note's gradient weight $σ (\overset{r}{^}_{l} - \overset{r}{^}_{w}) = σ (0.2) = 0.5498$ is bigger than half — DPO pushes hardest exactly here, on the mistake. Auto hard-example mining confirmed.

Worked example Example 4 — Cells D & E: the two limits

Fix $β = 0.2$ . Push the log-ratio gap $Δ ℓ = ℓ_{w} - ℓ_{l}$ to extremes.

Forecast: As $Δ ℓ \to + \infty$ does the loss go to $0$ or to $lo g 2$ ? As $Δ ℓ \to - \infty$ does it go to a finite number or to infinity?

Case D ( $Δ ℓ = + 30$ ): $m = 0.2 \cdot 30 = 6$ . $σ (6) = 0.99752$ , $L = - lo g (0.99752) = 0.002472$ . Why this step? A huge positive margin means the model is certain it's right; loss approaches the floor $0$ .
Case E ( $Δ ℓ = - 30$ ): $m = 0.2 \cdot (- 30) = - 6$ . $σ (- 6) = 0.002473$ , $L = - lo g (0.002473) = 6.0025$ . Why this step? A huge negative margin means the model is confidently wrong; loss grows roughly linearly like $- m$ .
Gradient behaviour: the gradient weight is $σ (- m)$ . In Case D that is $σ (- 6) = 0.00247$ — a dead gradient (saturation, nothing to learn). In Case E it is $σ (6) = 0.9975$ — a near-maximal gradient (learn hard). Why this step? This is why DPO self-balances: confident-correct pairs stop pulling, confident-wrong pairs pull almost fully.

Verify: For large $∣ m ∣$ , $- lo g σ (m) \approx lo g (1 + e^{- m}) \approx e^{- m}$ (Case D: $e^{- 6} = 0.00248 \approx 0.00247$ ✓) and $\approx - m$ (Case E: $- (- 6) = 6 \approx 6.0025$ ✓). Both limits confirmed.

Worked example Example 5 — Cell F: sweep

$β$ with the same policy deviation Hold the log-ratio gap fixed at $Δ ℓ = ℓ_{w} - ℓ_{l} = 3.0$ . Compute the loss for three leash strengths.

Forecast: As $β$ grows, does the loss for the same policy shift go up or down? What does that mean for how far the model is willing to move?

$β = 0.1$ : $m = 0.1 \cdot 3 = 0.3$ , $L = - lo g σ (0.3) = 0.5544$ .
$β = 0.5$ : $m = 0.5 \cdot 3 = 1.5$ , $L = - lo g σ (1.5) = 0.2014$ .
$β = 1.0$ : $m = 1.0 \cdot 3 = 3.0$ , $L = - lo g σ (3.0) = 0.04859$ . Why these steps? The same deviation $Δ ℓ$ produces a bigger reward margin when $β$ is bigger, so the sigmoid saturates and the loss shrinks.
Interpretation: a larger $β$ means the model becomes "satisfied" (low loss, small gradient) after a smaller actual change in $π_{θ}$ . So large $β$ = tight leash = less deviation from $π_{ref}$ .

Verify: Losses strictly decrease $0.554 > 0.201 > 0.049$ as $β$ rises. ✓ This is the exact counter-intuitive fact in the parent note's first "steel-manned mistake": bigger $β$ does not mean more freedom. See also KL Divergence — $β$ is the coefficient on the KL leash term.

Worked example Example 6 — Cell G: which way does each response actually move?

From the parent's gradient $\nabla_{θ} L = - β σ (\overset{r}{^}_{l} - \overset{r}{^}_{w}) (\nabla_{θ} lo g π_{θ} (y_{w}) - \nabla_{θ} lo g π_{θ} (y_{l})),$ take Example 2's numbers ( $\overset{r}{^}_{w} = - 0.05$ , $\overset{r}{^}_{l} = 0.15$ , $β = 0.1$ ).

Forecast: Will an update raise or lower the winner's probability? By a relatively big or small step?

Weight: $w = σ (\overset{r}{^}_{l} - \overset{r}{^}_{w}) = σ (0.15 - (- 0.05)) = σ (0.2) = 0.5498$ . Why this step? The weight scales the whole update; larger weight = larger step.
The bracket adds $+ \nabla lo g π_{θ} (y_{w})$ (push winner up) and subtracts $\nabla lo g π_{θ} (y_{l})$ (push loser down), with the outer $-$ arranged so gradient descent increases $lo g π_{θ} (y_{w})$ and decreases $lo g π_{θ} (y_{l})$ . Why this step? This is the "increase winner, decrease loser" behaviour — the whole point of preference training.
Compare with Example 1 (correct pair): weight $= σ (- 0.1 - 0.2) = σ (- 0.3) = 0.4256 < 0.5498$ . Why this step? The wrong pair (Ex 2) gets the larger update — automatic focus on mistakes.

Verify: Weight on the wrong pair $0.5498$ > weight on the correct pair $0.4256$ . ✓ Direction: winner up, loser down, exactly as claimed.

Worked example Example 7 — Cell H: a real batch (word problem)

A support chatbot is being aligned to prefer polite answers. We have a batch of three pairs with $β = 0.2$ . Each row gives $(ℓ_{w}, ℓ_{l})$ :

Pair 1 (polite vs curt): $(1.0, - 0.5)$
Pair 2 (helpful vs refusal): $(0.2, 0.4)$ — model currently likes the refusal!
Pair 3 (concise vs rambling): $(2.5, 0.5)$

Forecast: Which single pair dominates the batch loss? Guess before computing.

Margins: $m_{1} = 0.2 (1.0 - (- 0.5)) = 0.30$ ; $m_{2} = 0.2 (0.2 - 0.4) = - 0.04$ ; $m_{3} = 0.2 (2.5 - 0.5) = 0.40$ . Why this step? Only the per-pair margin feeds the loss; compute all three first.
Losses: $L_{1} = - lo g σ (0.30) = 0.5544$ ; $L_{2} = - lo g σ (- 0.04) = 0.7132$ ; $L_{3} = - lo g σ (0.40) = 0.5130$ . Why this step? Convert each margin to logistic loss.
Batch loss = mean $= (0.5544 + 0.7132 + 0.5130) /3 = 0.5935$ . Why this step? The DPO objective is an expectation (average) over the dataset.

Verify: Pair 2 has the only negative margin and the highest loss $0.7132 > lo g 2$ — it dominates and will receive the biggest gradient, correctly steering the bot away from refusing. ✓ Batch mean $0.5935$ sits between the min and max losses as any mean must.

Worked example Example 8 — Cell I: exam sign trap

Both log-ratios are negative: $ℓ_{w} = - 2.0$ , $ℓ_{l} = - 3.0$ , with $β = 0.4$ . A student panics: "the model made both responses less likely than the reference — surely it's broken and the loss is huge?"

Forecast: Is the loss above or below $lo g 2$ ? Is the model actually wrong here?

Rewards: $\overset{r}{^}_{w} = 0.4 \cdot (- 2.0) = - 0.8$ , $\overset{r}{^}_{l} = 0.4 \cdot (- 3.0) = - 1.2$ . Why this step? Absolute negativity does not matter — the loss only reads the difference.
Margin: $m = - 0.8 - (- 1.2) = 0.4$ . Positive! Why this step? Even though both dropped, the winner dropped less, so it is still ranked above the loser.
$L = - lo g σ (0.4) = 0.5130$ .

Verify: $m > 0 \Rightarrow L < lo g 2$ ; indeed $0.513 < 0.693$ . ✓ The panic was wrong. This is exactly why DPO uses the ratio to the SFT reference and only its difference — absolute likelihoods can both fall while the preference ranking stays correct (compare with Reward Modeling, where an explicit reward would also only be judged by differences).

Active recall

Recall Cover the answers and predict each cell

Which cell has loss exactly $lo g 2$ ? ::: Cell C — the tie / untrained start, $m = 0$ , for any $β$ . Sign of margin whenever the model is currently wrong? ::: Negative ( $m < 0$ ), giving $L > lo g 2$ . Two negative log-ratios — can the loss still be small? ::: Yes (Ex 8) — only the difference matters, so the winner just needs to drop less than the loser. As $β$ rises with fixed $Δ ℓ$ , loss goes...? ::: Down (Ex 5) — bigger margin, more saturation, tighter leash, less deviation. Where is the gradient dead vs maximal? ::: Dead at large positive margin (confident-correct, Ex 4D); near-maximal at large negative margin (confident-wrong, Ex 4E). Batch loss of Ex 7? ::: $0.5935$ , dominated by the one negative-margin pair.