Reinforcement Learning Based Fine-Tuning (RLHF vs. DPO)

Introduction

Large Language Models (LLMs) have transformed how we interact with AI. But training base models with next-token prediction (a.k.a. pretraining) is only half the story. To make models useful, safe, and aligned with human expectations, we fine-tune (FT) them. Among FT approaches, reinforcement learning (RL) based methods stand out for their ability to inject preferences, tone, and safety into otherwise raw models.

Two methods dominate the landscape: Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). Both aim to align LLMs with human intent, but they differ in complexity, data requirements, and operational trade-offs.

The Two Approaches

Reinforcement Learning from Human Feedback (RLHF)

Reinforcement Learning from Human Feedback (RLHF) is the most widely used alignment method today, powering models like InstructGPT and ChatGPT. The idea is that a pretrained model knows language, but not what humans want. Left alone, it can sound intelligent yet give unsafe, overly long, or unhelpful answers. RLHF fixes this by teaching the model from human preferences.

First, we collect examples where humans rank one response as better than another. A reward model is then trained to mimic these judgments, meaning it learns to assign a higher score to preferred responses. Finally, we use RL (often through Proximal Policy Optimization, or PPO) to adjust the language model’s policy so that the expected reward is maximized.

Mathematically, the model learns a policy $\pi_\theta(y|x)$ that increases

where $r(x,y)$ comes from the reward model. This loop teaches the model not just to be correct, but to sound helpful, safe, and human-like.

In other words, RLHF combines human preference data with RL to teach the model not just what is correct, but what humans actually prefer. By first training a reward model to act as a proxy for human judgment and then optimizing the base model against it, RLHF enables fine-grained control over behaviors like helpfulness, tone, refusal, and safety.

• Pipeline:
  
  • Train a reward model from preference data
  • Fine-tune the base model using RL (often PPO, Proximal Policy Optimization)
    
    
• Strengths:
  
  • Flexible: reward shaping can encode nuanced goals
  • Strong track record in models like InstructGPT and ChatGPT
    
    
• Weaknesses:
  
  • Complex: requires reward model training + RL loop
  • Expensive: unstable training, high compute demand
  • Harder to reproduce in smaller labs

Direct Preference Optimization (DPO)

Direct Preference Optimization (DPO) is a newer and more streamlined alternative to RLHF. Instead of training a separate reward model and running a full RL loop, DPO directly optimizes the language model using human preference pairs (examples where humans have labeled one response as “preferred” over another).

DPO works through a contrastive objective, meaning it compares two candidate outputs (preferred vs. rejected) and increases the likelihood of the preferred one while decreasing that of the rejected one. This creates a push-pull dynamic similar to how contrastive learning distinguishes positive and negative examples. In formal terms, the model learns to assign higher log-probabilities to preferred outputs relative to rejected ones, effectively aligning generation behavior with human preferences.

This design bypasses the need for PPO (Proximal Policy Optimization), the RL algorithm commonly used in RLHF. PPO stabilizes policy updates by constraining how much the new model’s behavior can deviate from the old one in a single step (using a clipping mechanism in the loss). While PPO is powerful, it requires iterative rollouts, reward models, and careful hyperparameter tuning, all of which make RLHF engineering-heavy.

DPO keeps the benefits of RLHF but removes the complicated RL loop. Instead of building a reward model and running PPO updates, DPO directly compares preferred and rejected responses. It optimizes a contrastive objective that increases the log-probability of preferred answers while decreasing that of rejected ones. This turns alignment into a simple supervised learning problem rather than a reinforcement one. 

The result: similar alignment quality, much less compute, and a method that’s easy for smaller teams to train, without sacrificing the core idea of learning from human preferences.

Said another way, the motivation behind DPO is to simplify this pipeline while keeping the benefits of preference alignment. By removing the instability of reward modeling and reinforcement training, DPO makes alignment much more accessible for smaller labs, startups, or research environments. In practice, DPO often achieves performance competitive with RLHF but with significantly less compute and complexity.

The table below can be helpful in understanding the difference between optimizing a contrastive objective vs a proximal policy.

• Pipeline:
  • Skip the reward model entirely
  • Directly optimize the policy (the model) on preference pairs via a contrastive objective
    
    
• Strengths:
  • Simpler, more stable than RLHF
  • Requires less compute
  • Easy to implement (a few lines of code vs. an RL loop)
    
    
• Weaknesses:
  • Less flexible and cannot shape rewards arbitrarily
    Works best when preferences are clean and consistent

Side-by-Side Comparison

Think of RLHF as training a judge (reward model) and then training the athlete (the LLM) with that judge’s feedback. DPO, in contrast, cuts out the middleman and trains the athlete directly from human votes.

When to Use Which?

Choosing between RLHF and DPO depends on your constraints and goals:

• Use RLHF when:
  • You need production-grade alignment (e.g., safety, tone, refusal behaviors)
  • You can afford the compute and engineering overhead
  • Reward shaping is critical (e.g., balancing helpfulness vs. harmlessness).
  • You want online control: models can be further trained with custom reward tweaks
    
    
• Use DPO when:
  • You want simplicity and fast iteration
  • You’re in a research or startup setting with limited compute
  • You just need to capture basic user preferences
  • You want an “80/20” alignment solution with minimal engineering
    

Evidence From Practice

• RLHF: Popularized by OpenAI’s InstructGPT (Ouyang et al., 2022). Still the de facto choice for industry-grade alignment
• DPO: Introduced in “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” (Rafailov et al., 2023). Now a hot research area, with stable results and easy adoption in open-source labs
  

Key Takeaways

• Both methods use human preferences to make models more aligned
• RLHF is flexible but costly. DPO is simple and practical
• For startups, academic labs, or quick iterations → DPO is often enough
• For big tech, safety-critical deployments, and nuanced control → RLHF remains dominant

Video Explanations

• This video titled, “Reinforcement Learning from Human Feedback” by IBM Technology, provides an excellent understanding behind the motivation and architecture of RLHF (Runtime: 11 mins)

• Aligning LLMs with DPO (Youtube, DeepLearningAI): Practical implementation of DPO with HuggingFace. It bridges the theory with hands-on training code, demonstrating how to replace PPO-based RLHF with a simpler DPO objective (Runtime: 58 mins)

Further Reading and Resources:

• Foundational Papers
  • Ouyang et al., 2022: Training language models to follow instructions with human feedback (Introduces RLHF, used in InstructGPT)
  • Rafailov et al., 2023: Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Introduces DPO)
  • Bai et al., 2022: Constitutional AI: Harmlessness from AI Feedback (Anthropic’s approach to safer alignment)
  • Ethayarajh et al., 2024: Kahneman-Tversky Optimization (KTO) (A newer alternative to DPO inspired by decision theory)
    

• Open-Source Implementations
  • Hugging Face TRL (Transformers Reinforcement Learning) Library: Ready-to-use RLHF, DPO, and PPO code
  • Hugging Face DPO Tutorial: Hands-on walkthrough of Direct Preference Optimization
  • TRL GitHub Repository: Official repo with examples
    

• Blog Posts & Guides
  • Hugging Face Blog: Fine-tuning LLMs with RLHF: Step-by-step RLHF guide
  • Alignment Handbook (open-source): Hugging Face toolkit for alignment research

Related Articles:

1. Introduction to Reinforcement Learning: A Beginner’s Guide
2. What does ‘policy’ in Reinforcement Learning mean?

Author: Roy Gabriel, PhD, Georgia Tech