Direct Preference Optimization (DPO): Question bank

Quick symbol refresher so no notation is unearned:

$x$ = the prompt; $y_{w}$ = the winner (human-chosen response); $y_{l}$ = the loser (rejected response). The subscripts $w$ and $l$ always mean "winner" and "loser".
$π_{θ}$ = the model we are training (starts as the SFT checkpoint).
$π_{ref}$ = a frozen copy of that same SFT checkpoint — never updated.
$\overset{r}{^}_{θ} (x, y) = β lo g \frac{π _{θ} ( y ∣ x )}{π _{ref} ( y ∣ x )}$ = the implicit reward: DPO's reward is not a separate network, it is this ratio. Shorthand: $\overset{r}{^}_{w} = \overset{r}{^}_{θ} (x, y_{w})$ and $\overset{r}{^}_{l} = \overset{r}{^}_{θ} (x, y_{l})$ .
$β$ = the scalar strength of the "leash" tying $π_{θ}$ to $π_{ref}$ .
$σ$ = the sigmoid, $σ (z) = \frac{1}{1 + e ^{- z}}$ , which turns a score difference into a probability (see figure below).
$Z (x) = \sum_{y} π_{ref} (y ∣ x) e^{r (x, y) / β}$ = the partition function: a normalizer that sums over every possible response to prompt $x$ . It is intractable (there are astronomically many sequences), and eliminating it is DPO's central trick (see the cancellation figure below).

Figure — Direct Preference Optimization (DPO)

The sigmoid above is the whole loss in one picture: the horizontal axis is the reward margin $\overset{r}{^}_{w} - \overset{r}{^}_{l}$ . When the winner already scores higher (margin positive, right side) $σ \to 1$ and the loss $- lo g σ$ shrinks toward $0$ ; when the model is wrong (margin negative, left side) the loss grows without bound. At margin $0$ the loss is exactly $lo g 2$ .

The second figure shows why $Z (x)$ disappears: both the winner's and loser's rewards carry the same $+ β lo g Z (x)$ block (because it depends only on the shared prompt $x$ ), so subtracting them (the Bradley-Terry difference) makes the blocks annihilate.

True or false — justify

TF1. DPO removes reinforcement learning entirely from the alignment pipeline.

True. DPO rewrites the RL objective as a single supervised classification loss on preference pairs — no policy rollouts, no PPO, no on-the-fly reward sampling.

TF2. Because DPO trains no reward model, there is no reward signal anywhere in it.

False. The reward is implicit:

\overset{r}{^}_{θ} = β lo g (π_{θ} / π_{ref})

. The policy itself parameterizes the reward — that's the entire trick, not the absence of reward.

TF3. Increasing

β

lets the model change more from its starting point.

False.

β

scales the reward gap for a given deviation, so the sigmoid saturates sooner and gradients vanish → the model deviates less. Large

β

= tighter leash.

TF4. The partition function

Z (x)

cancels no matter what preference model you use.

False. It cancels because Bradley-Terry takes a difference of rewards for the same prompt

x

, and

Z (x)

depends only on

x

. Change to a model without that same-prompt subtraction and

Z (x)

can return.

TF5. At initialization (

π_{θ} = π_{ref}

) the per-example loss is exactly

0

False. Both log-ratios are

0

, the margin is

0

, and the loss is

- lo g σ (0) = lo g 2 \approx 0.693

— the coin-flip baseline, not zero.

TF6. The frozen reference model is redundant and can be dropped to save memory.

False. Without the ratio to

π_{ref}

the loss rewards raw likelihood, collapsing onto a few high-frequency sequences (degeneration), and

Z (x)

no longer cancels.

TF7. DPO can learn from a single labeled response (not a pair).

False. The

lo g Z (x)

cancellation only happens when subtracting two rewards for the same

x

. One response leaves

Z (x)

intact and intractable — pairs are structurally required.

TF8. DPO needs to sample new responses from

π_{θ}

during training.

False. It uses the fixed preference dataset

(x, y_{w}, y_{l})

you already have. The removal of on-policy sampling is a major reason DPO is cheaper and more stable than PPO.

TF9. A large positive reward margin

\overset{r}{^}_{w} - \overset{r}{^}_{l}

means the model still has a lot to learn on that pair.

False. A large positive margin means the model already prefers the winner, so the loss and gradient are small — DPO has essentially finished with that easy pair.

TF10. Raising the absolute log-probability of

y_{w}

always lowers the DPO loss.

False. Only the ratio

lo g (π_{θ} / π_{ref})

matters. If

π_{ref}

already assigns

y_{w}

high probability, pushing raw likelihood up barely moves the ratio.

Spot the error

SE1. "DPO's loss is

- E [lo g σ (\overset{r}{^}_{w} + \overset{r}{^}_{l})]

The winner and loser rewards are subtracted, not added:

- E [lo g σ (\overset{r}{^}_{w} - \overset{r}{^}_{l})]

. Adding them destroys the preference direction and the

Z (x)

cancellation.

SE2. "The implicit reward is

\overset{r}{^}_{θ} = lo g π_{θ}

It must be the ratio to the reference and scaled by $β$ :

\overset{r}{^}_{θ} = β lo g (π_{θ} / π_{ref})

. Dropping

π_{ref}

turns it into raw likelihood and reintroduces degeneration.

SE3. "We can update

π_{ref}

alongside

π_{θ}

so the leash tightens over time."

π_{ref}

must stay frozen. If it tracks

π_{θ}

, the ratio drifts toward

1

, the reward signal collapses, and the KL anchor to the original SFT model is lost.

SE4. "DPO needs a value model like PPO does."

DPO needs no value/critic model. PPO-RLHF carries policy + reference + reward + value (four models); DPO keeps only policy + reference (two).

SE5. "The sigmoid in DPO is optional smoothing; you could use the raw margin as the loss."

The sigmoid comes directly from the Bradley-Terry preference probability. Replacing it changes what you're modeling and loses the bounded, well-behaved logistic gradient that weights hard examples.

SE6. "Since

lo g Z (x) - lo g Z (x) = 0

, we could ignore

Z (x)

even before subtracting."

You may only discard it after forming the difference. Before that,

Z (x) = \sum_{y} π_{ref} (y ∣ x) e^{r / β}

is a sum over all sequences — intractable, not ignorable.

SE7. "DPO minimizes the KL divergence directly as a separate term in its loss."

There is no explicit KL term in the DPO loss. The KL constraint was absorbed analytically into the closed-form optimal policy back in the derivation; the ratio-form reward is what carries it.

Why questions

WHY1. Why does DPO focus learning on the model's current mistakes automatically?

The gradient weight is

σ (\overset{r}{^}_{l} - \overset{r}{^}_{w})

, which is large exactly when the model wrongly rates the loser above the winner — an automatic hard-example weighting with no reward model needed.

WHY2. Why is the difference of rewards, not each reward alone, what DPO can compute?

Each individual reward carries the intractable

+ β lo g Z (x)

term; only in the difference (same

x

) does that term cancel, leaving a fully computable expression.

WHY3. Why does DPO start from an SFT checkpoint rather than a raw pretrained model?

π_{ref}

must be a competent instruction-follower so the ratio measures preference improvement, not basic fluency; SFT provides that sane, on-distribution anchor.

WHY4. Why is PPO-RLHF unstable in a way DPO avoids?

PPO optimizes against a learned, noisy reward with on-policy rollouts (susceptible to reward hacking and high variance); DPO replaces this with a fixed, deterministic supervised loss.

WHY5. Why does the same policy shift produce a smaller loss when

β

is larger?

A fixed log-ratio margin

Δ

becomes reward margin

β Δ

; larger

β

makes

σ (β Δ)

closer to

1

, so

- lo g σ

is smaller and the model is "satisfied" with less movement.

WHY6. Why can't we just supervise on

y_{w}

alone (plain SFT) and skip

y_{l}

SFT on

y_{w}

raises the winner's probability but never lowers the loser's; DPO's contrastive form explicitly pushes

y_{l}

down, encoding the relative human preference.

WHY7. Why is the loss at initialization exactly

lo g 2

and not something dataset-dependent?

At init

π_{θ} = π_{ref}

makes every margin

0

regardless of data, and

- lo g σ (0) = - lo g (1/2) = lo g 2

for every example.

Edge cases

EC1. What is the loss when the model confidently prefers the loser (large negative margin)?

σ

of a large negative number

\to 0

, so

- lo g σ \to + \infty

(grows without bound); the loss is large and the gradient strongly corrects it.

EC2. What happens to the gradient when the margin is very large and positive?

σ (\overset{r}{^}_{l} - \overset{r}{^}_{w}) \to 0

, so the gradient vanishes — that pair is already solved and contributes almost nothing to learning.

EC3. What if

y_{w}

and

y_{l}

are genuinely equally good (a mislabeled or tie pair)?

DPO still forces the winner up and loser down, injecting noise; ties or noisy labels degrade DPO because it assumes a real preference direction exists.

EC4. What if

π_{ref}

assigns zero probability to the chosen response

y_{w}

The log-ratio

lo g (π_{θ} / π_{ref})

blows up (division by ~0), destabilizing training — one reason the reference should share support with the data (SFT ensures this).

EC5. As

β \to 0

, what actually happens to learning?

The whole reward margin

β (Δ_{w} - Δ_{l})

is scaled toward

0

, so the sigmoid sits near

\frac{1}{2}

and the gradient magnitude

\to 0

— the model stops learning (signal starved), it does not drift freely.

EC6. As

β \to \infty

, what is the asymptotic behavior?

It splits by pair: already-correct pairs saturate and their gradients vanish, but any misordered pair gets a reward margin scaled arbitrarily large and negative, producing unboundedly large gradients — so behavior is not simply "no movement" but a sharp, unstable push to fix wrong pairs.