Lesson · 40 min · Free
Preference Alignment: DPO and RLHF
Preference Alignment: DPO and RLHF body { font-family: sans-serif; line-height: 1.6; color: #333; max-width: 900px; margin: 0 auto; padding: 20px; } h1, h2 { color: #0056b3; } pre { background-color: #f4f4f4; border: 1px
Preference Alignment: DPO and RLHF
Welcome to this module on Preference Alignment, a crucial step in fine-tuning Large Language Models (LLMs) to better suit human expectations and ethical guidelines. While foundational LLMs are incredibly powerful, they are often trained on vast, uncurated internet data. This can lead to responses that are unhelpful, untruthful, or even harmful. Preference alignment techniques address this by incorporating human feedback directly into the model's training process, guiding it to generate responses that are preferred by humans. For pharmacy and biotech students, understanding preference alignment is particularly important. Imagine an LLM designed to assist with drug discovery or patient information. An unaligned model might generate plausible-sounding but factually incorrect information about drug interactions, provide biased advice, or even hallucinate non-existent compounds. Preference alignment helps ensure that the LLM's outputs are not only grammatically correct but also accurate, safe, and aligned with professional and ethical standards.
Reinforcement Learning from Human Feedback (RLHF)
Reinforcement Learning from Human Feedback (RLHF) has been a groundbreaking technique in aligning LLMs. It involves a multi-step process that leverages human preferences to train a reward model, which then guides the LLM's fine-tuning. Let's break down the core components: Supervised Fine-Tuning (SFT): Initially, a pre-trained LLM is fine-tuned on a dataset of high-quality human-written demonstrations. This helps the model learn to follow instructions and generate coherent text. Reward Model Training: Human annotators are presented with multiple responses generated by the SFT model for a given prompt. They then rank these responses from best to worst based on criteria like helpfulness, truthfulness, and harmlessness. This human preference data is used to train a separate "reward model" (often another neural network). The reward model learns to predict a scalar "reward" for any given prompt-response pair, effectively quantifying how "good" a response is according to human preferences. Reinforcement Learning Fine-Tuning: The SFT model is then further fine-tuned using a Proximal Policy Optimization (PPO) algorithm. The reward model acts as the "environment" providing feedback (the reward score) to the LLM. The LLM learns to generate responses that maximize this reward, thereby aligning its outputs with human preferences. A critical component here is the KL divergence penalty, which prevents the model from deviating too far from its initial SFT behavior, preserving its general language capabilities while aligning it. While effective, RLHF can be computationally intensive and complex to implement. Training the reward model requires significant human annotation, and the PPO step itself can be challenging to stabilize. Here's a conceptual Python-like pseudocode illustrating the RLHF process: # Step 1: Supervised Fine-Tuning (SFT) # Model is fine-tuned on instruction-following data sft_model = fine_tune(base_llm, instruction_dataset) # Step 2: Reward Model Training # 2a: Generate responses for prompts using sft_model prompts = ["Explain protein folding.", "Write a short story about a scientist."] responses_A = [sft_model.generate(p) for p in prompts] responses_B = [sft_model.generate(p) for p in prompts] # Generate multiple options # 2b: Human annotation to rank responses human_preferences = [ {"prompt": p, "preferred": resp_A, "less_preferred": resp_B} for p, resp_A, resp_B in zip(prompts, responses_A, responses_B) # In reality, this is a much more complex human labeling process ] # 2c: Train a reward model based on human preferences reward_model = train_reward_model(human_preferences) # Step 3: Reinforcement Learning Fine-Tuning (PPO) # Initialize policy and value networks from sft_model ppo_agent = PPOAgent(sft_model) for episode in range(num_ppo_epochs): # Generate responses using the current policy (LLM) generated_responses = ppo_agent.generate_responses(prompts) # Get rewards from the reward model rewards = reward_model.predict_rewards(prompts, generated_responses) # Update the LLM policy using PPO, maximizing rewards ppo_agent.update_policy(prompts, generated_responses, rewards) final_aligned_llm = ppo_agent.get_llm()
Direct Preference Optimization (DPO)
Direct Preference Optimization (DPO) emerged as a simpler and more stable alternative to RLHF. It achieves similar alignment goals without explicitly training a separate reward model or using complex reinforcement learning algorithms like PPO. DPO directly optimizes the language model policy to satisfy human preferences. The core idea behind DPO is to reframe the preference alignment problem as a classification task. Instead of training a reward model to assign scores, DPO directly optimizes the LLM to maximize the probability of generating preferred responses while minimizing the probability of generating dispreferred responses, based on a dataset of human preferences. This dataset is similar to the one used for training the reward model in RLHF: for each prompt, there's a preferred response and one or more dispreferred responses. DPO uses a single, modified loss function that directly incorporates the human preference pairs. This loss function is derived from the theoretical connection between reward models and optimal policies in RL. By optimizing this loss, the LLM implicitly learns a reward function that aligns with human preferences. The advantages of DPO include: Simplicity: No need to train a separate reward model. Stability: Avoids the challenges of training PPO, which can be sensitive to hyperparameters. Computational Efficiency: Can be less resource-intensive than full RLHF. Here's a conceptual Python-like pseudocode for DPO: import torch import torch.nn.functional as F # Assume we have an SFT model (fine-tuned base LLM) sft_model = load_sft_model() dpo_model = sft_model # DPO directly fine-tunes this model # Preference dataset: List of dictionaries like {"prompt": ..., "chosen": ..., "rejected": ...} preference_dataset = [ {"prompt": "Explain CRISPR.", "chosen": "CRISPR is a gene-editing tool...", "rejected": "CRISPR is a new type of coffee."}, {"prompt": "Drug interaction with Warfarin?", "chosen": "Warfarin interacts with many drugs, including...", "rejected": "Warfarin is a safe drug with no interactions."}, # ... more human preference pairs ] optimizer = torch.optim.Adam(dpo_model.parameters(), lr=1e-5) for epoch in range(num_dpo_epochs): for item in preference_dataset: prompt = item["prompt"] chosen_response = item["chosen"] rejected_response = item["rejected"] # Calculate log probabilities for chosen and rejected responses # This involves passing (prompt + chosen) and (prompt + rejected) through the model # and getting the log-likelihood of the response tokens given the prompt. log_prob_chosen = dpo_model.get_log_probability(prompt, chosen_response) log_prob_rejected = dpo_model.get_log_probability(prompt, rejected_response) # Calculate the DPO loss # beta is a hyperparameter that controls the strength of the preference beta = 0.1 loss = -F.logsigmoid(beta * (log_prob_chosen - log_prob_rejected)) # Backpropagate and update model parameters optimizer.zero_grad() loss.backward() optimizer.step() print("DPO fine-tuning complete!") In the DPO loss function, log_prob_chosen - log_prob_rejected represents the log-odds of the chosen response being better than the rejected one. The -F.logsigmoid(...) part ensures that we are maximizing this log-odds difference, effectively making the model prefer the chosen response. The beta parameter scales the preference strength.
Key Takeaways
Purpose of Alignment: To make LLM outputs helpful, truthful, and harmless, especially critical in fields like pharmacy/biotech where accuracy and safety are paramount. RLHF Process: Involves Supervised Fine-Tuning, training a Reward Model from human preferences, and then fine-tuning the LLM using Reinforcement Learning (PPO) guided by the reward model. DPO Simplicity: Direct Preference Optimization (DPO) is a more straightforward alternative to RLHF, directly optimizing the LLM using a single loss function derived from human preference pairs, without needing a separate reward model or complex RL. Human Feedback is Key: Both methods fundamentally rely on human feedback to define what constitutes a "good" or "bad" response. Applications in Pharmacy/Biotech: Ensures LLMs provide accurate drug information, ethical advice, and reliable research insights, minimizing risks associated with unaligned models.
Practice Exercise
Imagine you are developing an LLM to assist pharmacists in providing patient counseling for new prescriptions. You have collected a dataset of prompts (e.g., "What are the side effects of Metformin?") and several generated responses, along with human rankings indicating which response is preferred and why (e.g., one response is more comprehensive and uses simpler language, while another is technically accurate but too jargon-filled). Describe how you would apply either RLHF or DPO to fine-tune this LLM. Specifically, outline the steps you would take, highlighting any unique considerations for a pharmaceutical context (e.g., emphasis on factual accuracy, clarity, avoiding alarmist language).
Watch the full lesson — free
This topic is part of The Complete LLM Engineering Bootcamp, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →