Direct Preference Optimization (DPO)

The setup (WHAT are the pieces)

DERIVATION from first principles

We derive DPO in three moves. Follow every "Why this step?".

Step 1 — The RLHF objective

RLHF maximizes expected reward while staying close to the reference model:

$\max_{\pi_\theta}\ \mathbb{E}_{x,\,y\sim\pi_\theta}\big[r(x,y)\big]\;-\;\beta\,\mathrm{KL}\!\big(\pi_\theta(y|x)\,\|\,\pi_{\text{ref}}(y|x)\big)$

Why this step? The reward term pushes toward high-quality answers; the KL term is a leash preventing the model from producing gibberish that games the reward (reward hacking).

Step 2 — Solve for the OPTIMAL policy

This objective has a known closed-form optimum (a standard result in KL-regularized reward maximization):

$\pi^*(y|x) \;=\; \frac{1}{Z(x)}\,\pi_{\text{ref}}(y|x)\,\exp\!\Big(\tfrac{1}{\beta}\,r(x,y)\Big)$

where $Z(x)=\sum_y \pi_{\text{ref}}(y|x)\exp(\tfrac1\beta r(x,y))$ is the partition function (normalizer).

Why this step? To turn "given reward, best policy" into "given policy, implied reward." $Z(x)$ is intractable (sums over all sequences) — that's the villain we must eliminate.

Step 3 — Invert to express the reward via the policy

Take logs of Step 2 and solve for $r$ :

$r(x,y) \;=\; \beta\,\log\frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} \;+\; \beta\,\log Z(x)$

Why this step? Now the reward is written in terms of the policy. The messy $Z(x)$ term depends only on $x$ , not on $y$ — remember that.

Step 4 — Plug into the Bradley–Terry preference model

Human preferences are modeled with Bradley–Terry: the probability that $y_w$ beats $y_l$ is

$P(y_w \succ y_l\mid x) = \sigma\big(r(x,y_w) - r(x,y_l)\big)$

where $\sigma$ is the sigmoid. Substitute the reward from Step 3:

$r(x,y_w)-r(x,y_l) = \beta\log\frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)} + \underbrace{\beta\log Z(x) - \beta\log Z(x)}_{=\,0}$

Why this step? This is the magic. Because we take a difference of rewards for the same prompt $x$ , the intractable $\log Z(x)$ cancels. No reward model, no partition function.

Step 5 — Write the loss

Replace $\pi^*$ with our trainable $\pi_\theta$ and maximize the log-likelihood of observed preferences (equivalently minimize):

Formula The DPO loss

$\mathcal{L}_{\text{DPO}} = -\,\mathbb{E}_{(x,y_w,y_l)}\left[\log\sigma\!\Big(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} \;-\; \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\Big)\right]$

Define the implicit reward $\hat r_\theta(x,y)=\beta\log\dfrac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ . Then simply $\mathcal{L}=-\mathbb E[\log\sigma(\hat r_w-\hat r_l)]$ — a binary classification loss.

Figure — Direct Preference Optimization (DPO)

HOW does the gradient behave? (intuition check)

The gradient is:

$\nabla_\theta\mathcal{L} = -\beta\,\mathbb{E}\Big[\underbrace{\sigma(\hat r_l-\hat r_w)}_{\text{weight}}\big(\nabla_\theta\log\pi_\theta(y_w|x) - \nabla_\theta\log\pi_\theta(y_l|x)\big)\Big]$

Worked examples

Worked example Example 1 — Compute a single loss term

Suppose $\beta=0.1$ . For one pair we have log-prob ratios: $\log\frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)} = 2.0$ , and $\log\frac{\pi_\theta(y_l)}{\pi_{\text{ref}}(y_l)} = -1.0$ .

Implicit rewards: $\hat r_w = 0.1\cdot 2.0 = 0.2$ , $\hat r_l = 0.1\cdot(-1.0)=-0.1$ . Why this step? Multiplying the log-ratio by $\beta$ converts a policy shift into a reward scale.

Margin $= 0.2-(-0.1)=0.3$ . Loss $=-\log\sigma(0.3) = -\log(0.574)=0.555$ . Why this step? Positive margin ⇒ model already prefers the winner ⇒ small-ish loss. If margin were negative, loss would exceed $\log 2\approx 0.693$ .

Common mistakes (Steel-manned)

Active recall

Recall Test yourself (reveal after answering)

Why does the partition function $Z(x)$ disappear?
What is the "implicit reward" in DPO?
What does increasing $\beta$ do to model deviation?
What loss value occurs at initialization ( $\pi_\theta=\pi_{\text{ref}}$ )?
Which four models does PPO-RLHF need that DPO replaces with two?

Answers: 1) It only depends on $x$ ; taking a difference of rewards for the same $x$ cancels it. 2) $\hat r_\theta=\beta\log(\pi_\theta/\pi_{\text{ref}})$ . 3) Reduces deviation (tighter KL leash). 4) $\log 2\approx0.693$ . 5) PPO needs policy+reference+reward+value; DPO needs only policy+reference.

Recall Feynman: explain to a 12-year-old

Imagine teaching a robot to answer nicely. The old way: hire a judge robot to score every answer, then train the answerer to chase high scores — clumsy and the answerer learns to trick the judge. DPO says: skip the judge! Just show the robot pairs — "this answer is good, this one is bad" — and nudge it to make the good one more likely and the bad one less likely, but only compared to how it used to talk, so it doesn't go crazy. The robot ends up being its own judge. Simple, and no cheating.

Connections

RLHF - Reinforcement Learning from Human Feedback — the parent method DPO simplifies.
PPO - Proximal Policy Optimization — the RL algorithm DPO removes.
Bradley-Terry Model — the preference likelihood used in Step 4.
KL Divergence — the regularizer whose closed-form optimum starts the derivation.
Reward Modeling — becomes implicit in DPO.
Supervised Fine-Tuning (SFT) — provides both $\pi_\theta$ init and $\pi_{\text{ref}}$ .
Sigmoid and Logistic Loss — DPO loss is just logistic regression on reward margins.

DPO stands for

Direct Preference Optimization

Two models DPO trains/uses

The trainable policy

\pi_\theta

and the frozen reference

\pi_{\text{ref}}

(the SFT checkpoint)

DPO loss formula

-\mathbb{E}[\log\sigma(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)})]

Why does the partition function Z(x) cancel in DPO

Because the Bradley-Terry model uses a difference of rewards for the same prompt x, and

\beta\log Z(x)

depends only on x

DPO implicit reward

\hat r_\theta(x,y)=\beta\log\frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}

Effect of increasing β in DPO

Tightens the KL leash → the policy deviates LESS from the reference model

DPO loss at initialization (π_θ = π_ref)

\log 2 \approx 0.693

per example (coin-flip baseline)

Closed-form optimal RLHF policy

\pi^*(y|x)=\frac{1}{Z(x)}\pi_{ref}(y|x)\exp(\frac1\beta r(x,y))

Preference model DPO assumes

Bradley-Terry:

P(y_w\succ y_l)=\sigma(r(x,y_w)-r(x,y_l))

What data does DPO need

Triples

(x, y_w, y_l)

: prompt, chosen (winning) response, rejected (losing) response

Why keep the reference model

Its ratio prevents likelihood collapse/degeneration and enables Z(x) cancellation; dropping it rewards raw high-frequency text

Which RLHF component does DPO eliminate

The separate reward-model training stage and the PPO/RL optimization loop

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, RLHF ka purana tareeka thoda painful hai: pehle ek reward model train karo jo har answer ko score de, phir PPO (ek RL algorithm) se policy ko us score ke peeche bhagao, aur saath mein KL penalty lagao taaki model gibberish na banane lage. Iske liye ek saath 4 models memory mein chahiye aur PPO tuning karna sar-dard hai. DPO bolta hai — yeh sab drama chhodo!

DPO ka core insight yeh hai: KL-regularized reward maximization ka ek closed-form optimal solution hota hai. Us formula ko ulta karke reward ko policy ke terms mein likh lo: $\hat r = \beta\log(\pi_\theta/\pi_{ref})$ . Ab jab hum Bradley-Terry preference model mein winner aur loser ke reward ka difference lete hain, toh woh dushman jaisa partition function $Z(x)$ automatically cancel ho jaata hai (kyunki dono same prompt $x$ ke hain). Result: ek simple logistic (sigmoid) loss — bas winner $y_w$ ki likelihood badhao aur loser $y_l$ ki ghatao, reference model ke comparison mein.

$\beta$ ko galat mat samjho — yeh brake hai, accelerator nahi. Bada $\beta$ matlab model reference se kam hatega (tight leash); chhota $\beta$ matlab zyada freedom. Aur ek cheez yaad rakhna: reference model ko mat hatao — uska ratio hi model ko degenerate hone se rokta hai aur $Z(x)$ ko cancel karne mein help karta hai.

Kyun important hai? Kyunki DPO ne alignment ko itna simple aur stable bana diya — bas preference pairs ka data lo aur ek supervised loss chala do, koi reward model nahi, koi RL nahi. Isiliye aaj-kal open-source models (jaise Zephyr, Llama variants) DPO se align hote hain. 80/20 rule: yeh derivation (Z cancel + implicit reward) samajh gaye toh poora DPO samajh gaye.

WHY does DPO exist?

The setup (WHAT are the pieces)

DERIVATION from first principles

Step 1 — The RLHF objective

Step 2 — Solve for the OPTIMAL policy

Step 3 — Invert to express the reward via the policy

Step 4 — Plug into the Bradley–Terry preference model

Step 5 — Write the loss

HOW does the gradient behave? (intuition check)

Worked examples

Common mistakes (Steel-manned)

Active recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Alignment, Prompting & RAG

Connections