Exercises — Direct Preference Optimization (DPO)

• $x$ = the prompt.
• $y_w$ = the winning (chosen / preferred) response.
• $y_l$ = the losing (rejected) response.
• ${\pi }_{\theta }$ is the policy being trained — it receives gradients.
• ${\pi }_{\text{ref}}$ is the frozen reference (the Supervised Fine-Tuning (SFT) checkpoint) — never updated.

Both log-ratios are $\log 1=0$, so ${\hat{r}}_w={\hat{r}}_l=0$, margin $=0$, and $\ell =-\log \sigma (0)=-\log (0.5)=\log 2\approx 0.693$. This is the coin-flip baseline: before any training the model has no preference, so it scores exactly $\log 2$ per example.

The margin is negative, meaning the model gives the loser a higher implicit reward than the winner — the model is wrong on this pair. Since $-\log \sigma$ is decreasing in $z$ (see figure), any $z<0$ gives loss above $\log 2$.

• ${\hat{r}}_w=0.2\times 1.5=0.3$.
• ${\hat{r}}_l=0.2\times (-0.5)=-0.1$.
• margin $z=0.3-(-0.1)=0.4$.
• $\sigma (0.4)=\frac{1}{1+e^{-0.4}}\approx 0.5987$.
• $\ell =-\log (0.5987)\approx 0.513$.

Positive margin ⇒ loss below $\log 2$: the model already leans the right way.

The reward margin is $z=\beta \Delta$.

• $\beta =0.1$: $z=0.2$, $\sigma (0.2)\approx 0.5498$, $\ell \approx 0.598$.
• $\beta =0.5$: $z=1.0$, $\sigma (1.0)\approx 0.7311$, $\ell \approx 0.313$.

Larger $\beta$ turns the same policy shift into a bigger reward margin, saturating the sigmoid → smaller loss → the model is "satisfied" and deviates less. Bigger $\beta$ = tighter leash.

$\ell =\log 2$ happens only when $\sigma (z)=0.5$, i.e. $z=0$. Since $z=\beta \Delta$ and $\beta \ne 0$, we need $\Delta =0$: the winner and loser have identical policy shifts. This is exactly the initialization / indifference case.

$\hat{r}(x,y_w)-\hat{r}(x,y_l)=\beta \log \frac{\pi (y_w|x)}{{\pi }_{\text{ref}}(y_w|x)}+\beta \log Z(x)-\left[\beta \log \frac{\pi (y_l|x)}{{\pi }_{\text{ref}}(y_l|x)}+\beta \log Z(x)\right].$ The two $\beta \log Z(x)$ terms are identical because $Z(x)$ depends only on $x$, and both responses share the same $x$. They subtract to $0$: $=\beta \log \frac{\pi (y_w|x)}{{\pi }_{\text{ref}}(y_w|x)}-\beta \log \frac{\pi (y_l|x)}{{\pi }_{\text{ref}}(y_l|x)}.$ Condition: both responses must be conditioned on the same prompt $x$. That is why DPO needs pairs per prompt — Bradley-Terry Model uses a difference of scores.

Deriving the weight. We need $\frac{d}{dz}[-\log \sigma (z)]$. Two facts:

• $\frac{d}{dz}\sigma (z)=\sigma (z)(1-\sigma (z))$ (standard sigmoid derivative).
• $\frac{d}{dz}\log \sigma (z)=\frac{{\sigma }^{\prime }(z)}{\sigma (z)}=\frac{\sigma (z)(1-\sigma (z))}{\sigma (z)}=1-\sigma (z)$.

Therefore $\frac{d\ell }{dz}=-(1-\sigma (z))=-\sigma (-z),$ using the symmetry $1-\sigma (z)=\sigma (-z)$. By the chain rule the parameter gradient is $\frac{d\ell }{dz}\cdot {\nabla }_{\theta }z$, and ${\nabla }_{\theta }z=\beta ({\nabla }_{\theta }\log {\pi }_{\theta }(y_w|x)-{\nabla }_{\theta }\log {\pi }_{\theta }(y_l|x))$. So the multiplying factor (the "weight") is exactly $\sigma (-z)=\sigma ({\hat{r}}_l-{\hat{r}}_w)$. Intuition: $1-\sigma (z)$ is "how much probability the model is missing" on getting the winner right — a bigger miss means a bigger push.

Evaluating:

• (a) badly wrong: ${\hat{r}}_l-{\hat{r}}_w=+4$, weight $=\sigma (4)\approx 0.982$ — large.
• (b) confidently right: ${\hat{r}}_l-{\hat{r}}_w=-4$, weight $=\sigma (-4)\approx 0.018$ — tiny.

The weight is large exactly when the model ranks the loser above the winner (a mistake), and near zero when the model is already correct. DPO automatically focuses on hard/wrong examples — a built-in hard-example weighting, no explicit Reward Modeling needed.

• Raw version: $\beta \log {\pi }_{\theta }=\beta \log 0.95$, a large-ish absolute value driven mostly by the token already being common — the model gets "credit" for probability it didn't create.
• Ratio version: $\beta \log \frac{0.95}{0.9}=\beta \log (1.056)\approx \beta (0.0541)$ — a small reward, because the model barely changed anything.

The ratio rewards only the change the policy made over the reference. This is what prevents collapse onto high-frequency generic text (degeneration) — the failure the raw version invites.

• $-\log \sigma (0.4)=-\log (0.5987)\approx 0.5130$.
• $-\log \sigma (-0.2)=-\log (0.4502)\approx 0.7981$.
• $-\log \sigma (1.0)=-\log (0.7311)\approx 0.3133$.
• Mean $=\frac{0.5130+0.7981+0.3133}{3}\approx 0.5415$.

Notice pair 2 (negative margin, a mistake) dominates the loss — consistent with the L3.2 hard-example weighting.

We need $z=\beta \Delta =1.0$ with $\Delta =4.0$, so $\beta =1.0/4.0=0.25$. If instead $\beta =2.5$ (ten times larger), then for the same $\Delta =4.0$, $z=10.0$, deep in the saturated region ($\sigma (10)\approx 0.99995$). The gradient nearly vanishes, so the model stops moving early → it deviates much less from ${\pi }_{\text{ref}}$. Larger $\beta$ = stronger leash, as in PPO - Proximal Policy Optimization's KL penalty analogue.

Take the natural log of both sides: $\log {\pi }^{\ast }=-\log Z(x)+\log {\pi }_{\text{ref}}+\frac{1}{\beta }r(x,y).$ Isolate $\frac{1}{\beta }r$: $\frac{1}{\beta }r(x,y)=\log {\pi }^{\ast }-\log {\pi }_{\text{ref}}+\log Z(x)=\log \frac{{\pi }^{\ast }}{{\pi }_{\text{ref}}}+\log Z(x).$ Multiply through by $\beta$: $r(x,y)=\beta \log \frac{{\pi }^{\ast }(y|x)}{{\pi }_{\text{ref}}(y|x)}+\beta \log Z(x).■$ This is the bridge: the reward is now written in terms of the policy, which is what lets us delete the separate reward model.

Using $\sigma (z)=\frac{1}{1+e^{-z}}$, so $\ell (z)=\log (1+e^{-z})$:

• $z\to +\infty$: $e^{-z}\to 0$, so $\ell \to \log 1=0$. The model confidently prefers the winner → no loss, no gradient. (Fully learned pair.)
• $z=0$: $\ell =\log 2\approx 0.693$. The indifference / initialization baseline.
• $z\to -\infty$: $e^{-z}$ dominates, $\ell \approx \log (e^{-z})=-z\to +\infty$, growing linearly in $|z|$. Confidently wrong pairs incur unbounded, linearly-growing loss — DPO punishes confident mistakes hard but not explosively (linear, not exponential), keeping training stable.

If $y_w=y_l$ then ${\pi }_{\theta }(y_w\mid x)={\pi }_{\theta }(y_l\mid x)$ and likewise for ${\pi }_{\text{ref}}$, so ${\hat{r}}_w={\hat{r}}_l$ exactly. Margin $z=0$, loss $=\log 2$, for every $\beta$ and every parameter setting. Why the gradient is zero. From L3.2 the parameter gradient of a pair is ${\nabla }_{\theta }\ell =-\beta \sigma ({\hat{r}}_l-{\hat{r}}_w)({\nabla }_{\theta }\log {\pi }_{\theta }(y_w\mid x)-{\nabla }_{\theta }\log {\pi }_{\theta }(y_l\mid x)).$ Here two things both hit zero. First, the weight is $\sigma ({\hat{r}}_l-{\hat{r}}_w)=\sigma (0)=0.5$ (finite). Second, and decisively, since $y_w=y_l$ the two log-prob terms are the same function, so their gradients are identical and ${\nabla }_{\theta }\log {\pi }_{\theta }(y_w\mid x)-{\nabla }_{\theta }\log {\pi }_{\theta }(y_l\mid x)=0$. The product is $-\beta \cdot 0.5\cdot 0=0$. So the pair contributes no learning signal but still counts toward the batch loss, dragging the reported average toward $\log 2$ and wasting compute. Such pairs should be filtered out.

Both margins equal ${\hat{r}}_w-{\hat{r}}_l=0.3$, so both give the same loss $\ell =-\log \sigma (0.3)=-\log (0.5744)\approx 0.554,$ and the same gradient weight $\sigma ({\hat{r}}_l-{\hat{r}}_w)=\sigma (-0.3)\approx 0.4256$. Pair A gives $\sigma (2.7-3.0)=\sigma (-0.3)$ and Pair B gives $\sigma (0.0-0.3)=\sigma (-0.3)$ — literally the same input, hence identical. Why DPO treats them the same. The loss depends only on the difference ${\hat{r}}_w-{\hat{r}}_l$, never on the absolute reward levels $3.0$ and $2.7$ versus $0.3$ and $0.0$. This is by design: absolute $\hat{r}$ values carry the arbitrary $\beta \log Z(x)$ offset that cancels in the difference (L3.1 / L4.3). So only the relative preference strength matters — exactly what the Bradley–Terry model encodes. Pair A's larger absolute rewards are completely invisible to the objective; a pair is "learned" based on how much the winner beats the loser, not on how large either raw score is.

$\ell (z)=-\log \sigma (z)$ is a strictly decreasing function of $z$ (since $\sigma$ is strictly increasing and $-\log$ is decreasing). At $z=0$, $\ell =\log 2$. Therefore:

• $z>0⟺\ell (z)<\ell (0)=\log 2$ (model prefers winner ⇔ loss below baseline).
• $z<0⟺\ell (z)>\log 2$ (model prefers loser ⇔ loss above baseline).

The equivalence is exact because $\ell$ is strictly monotone, so it crosses $\log 2$ exactly once, at $z=0$. $■$

1. What margin gives loss exactly $\log 2$?
2. As $z\to -\infty$, how fast does the loss grow?
3. Why is the gradient weight large on wrong examples?
4. Does DPO loss depend on absolute reward or only the difference?
5. What happens to deviation when $\beta$ increases?

Answers: 1) $z=0$. 2) Linearly, as $-z$. 3) Weight $=\sigma ({\hat{r}}_l-{\hat{r}}_w)$, near 1 when the loser is ranked above the winner. 4) Only the difference ${\hat{r}}_w-{\hat{r}}_l$. 5) Deviation decreases (tighter leash).