
Source: Paper titled “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” by Rafael et al
Direct Preference Optimization: RLHF Without the Headache
Imagine you ask a chatbot:
Prompt : “Reply politely to this customer complaint about a late delivery.”
The chatbot produces two candidate responses:
- Response A (, chosen): “I’m sorry your order arrived late. That’s not the experience we want you to have. If you share your order number, I’ll look into what happened and help make it right.”
- Response B (, rejected): “Delays happen sometimes. Check the tracking next time.”
Most people prefer A over B.
This is the core signal behind preference alignment: for the same prompt , we want the model to assign higher probability to the preferred response than to the rejected response .
If you have looked into preference alignment, you have probably encountered RLHF (Reinforcement Learning from Human Feedback). A common RLHF recipe looks like this:
- Train a reward model so that it scores chosen responses higher than rejected ones.
- Use an RL(reinforcement learning) algorithm (often PPO, Proximal Policy Optimization ) to update the chatbot policy so it generates outputs with higher reward scores while staying close to a reference model to avoid drifting.
RLHF works, but it can be heavy and finicky, two large models, multiple training stages, and stability issues.
Direct Preference Optimization (DPO) is a clean alternative. It keeps the core goal of RLHF, makes the model prefer answers humans prefer, but removes the most cumbersome piece: you do not train a separate reward model. Instead, DPO learns directly from preference pairs, and it tends to be surprisingly stable in practice.
What DPO is (and isn’t)
DPO is not reinforcement learning in the classic sense. There is no environment, no trajectories, and no policy rollouts. Training looks much closer to supervised learning.
What DPO does have is:
- A trained model:
- A frozen reference model: (often is the supervised fine-tuning model)
- Preference data: prompts with a chosen response and a rejected response.
The reference model acts like an anchor: you do want the policy to move toward preferred behavior, but you generally do not want it to drift arbitrarily far away from the starting behavior distribution (which can hurt fluency, truthfulness, or general capabilities). The reference model provides a stable baseline.
The core idea: improve the chosen-vs-rejected gap
A language model does not output a single “score” for an answer. Instead, it generates text, one token at a time, assigning a probability to each next token given the prompt and the tokens so far. If we want to compare two full responses, we need a way to turn “token-by-token probabilities” into one number per response.
A standard trick is to score a response by the sum of its token log-probabilities under the model. You can think of this as: “How likely is the model to produce this entire response?”, we denote that score as . Higher score means the model finds the response more likely.
Now we can define the model’s preference gap between the chosen and rejected responses:
If is large, the model clearly prefers the chosen response.
But here is the key: DPO does not only ask “does the model prefer ?” It asks: Does the model prefer more than the reference model prefers it?
So we also compute the same gap under the frozen reference:
Then we measure improvement over reference:
- If > 0, your model prefers over more than the reference does.
- If < 0, the model is not yet matching the preference signal relative to the baseline
From improvement Δ to a trainable loss

Source: Cameron R. Wolfe, Ph.D blog
Up to now, is just a score that tells us whether the current model agrees with the preference more than the reference model does:
- > 0: moving in the right direction
- < 0: needs correction
The remaining question is: how do we turn this single number into an objective we can optimize with gradient descent?
Step 1: Treat Δ as a “margin” and map it to a probability
A common modeling choice in preference learning is the probability that chosen beats rejected should increase smoothly as the margin increases. The simplest way to do that is using the Bradley-Terry model:
This behaves exactly the way we want:
- If = 0, then = 0.5: the model is indifferent.
- If > 0, then > 0.5: the model predicts the chosen response should win.
- If < 0, then < 0.5: the model is leaning the wrong way.
The hyperparameter controls how sharp this preference is:
- Larger makes change faster (stronger push).
- Smaller makes it change slower (gentler updates).
Step 2: Use the standard “chosen wins” log-loss
At this point we have a probability that the chosen response should beat the rejected one:
Since every training example says “ is preferred,” we can train this exactly like a binary classifier: maximize the probability of the correct label. The standard objective is log-loss (cross-entropy), i.e., the negative log of that probability:
Substituting the logistic preference probability gives the per-example DPO loss:
Finally, we average over the dataset:
If you expand , you recover the full expression:
Why this loss “does the right thing”
- When is large and positive, 1, so the loss 0. The model already agrees with the preference, so little update is needed.
- When is negative, is small, so the loss becomes large. Gradient descent will push upward, meaning it increases the relative probability of and/or decreases that of , compared to the reference.
In one sentence: DPO treats as a preference margin, turns it into a probability with a sigmoid, and applies standard log-loss to make “chosen wins” increasingly likely.
Why DPO often feels easier than RLHF
- RLHF trains a reward model; DPO does not
- In many RLHF pipelines, you first train a separate reward model to predict human preferences, and that model can be large (often built from the same backbone family as the policy). Training and maintaining this extra model adds compute, memory, and engineering work.
- DPO avoids this step by not learning an explicit reward model. Instead, it uses preference pairs directly in the loss through quantities we can compute from the language model itself: and (and the same scores under a frozen reference model). The key difference is that the reference model is not trained, it is only used to compute log-probabilities, so you are not maintaining a second trainable network like .
- RLHF uses an RL optimization loop; DPO looks like standard fine-tuning
- PPO-style RLHF typically requires generating rollouts, computing advantages, and tuning stabilization mechanisms (like clipping and KL control). This adds moving parts and can make training sensitive to hyperparameters.
- DPO training is closer to standard fine-tuning: you optimize a single differentiable loss on a fixed dataset of preference pairs, which tends to be simpler to run.
Summary
DPO gives you the alignment “direction” of RLHF, increasing the likelihood of preferred answers without the heavy machinery of training a reward model and running reinforcement learning. You train one model , anchor it with a frozen policy , and optimize a simple logistic loss over preference pairs. The result is a method that is often easier to implement, easier to scale, and easier to keep stable, while staying closely connected to the principles that made RLHF work in the first place.
Video Explanation and Paper
- “Direct Preference Optimization (DPO) – How to fine-tune LLMs directly without reinforcement learning” (Serrano.Academy, ~20 min): A clear, math-forward walkthrough that builds the DPO objective step-by-step from preference pairs, which I highly recommend because it explains every component used in DPO and how they fit together.
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model | DPO paper explained” (channel: AI Coffee Break with Letitia, ~8 min):A concise, high-level bridge from RLHF to DPO that explains the connection between them and provides the key background context for why DPO works.

