Your AI Knows How to Answer. But Who Teaches It What a Good Answer Is?

Rijul is developing LiveReview, an AI code review tool designed to avoid "blast-radius" issues in business-critical systems. He invites developers to explore the project, test it out, and provide feedback to enhance the product. Users interact with AI models like ChatGPT and Gemini, often receiving satisfactory responses. However, the question arises: how are these models trained to produce optimal answers? Humans play a role in this process.

There are two primary methods to teach language models to generate preferred responses: Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF). Both techniques employ human preferences to guide the model, but they approach the problem differently.

Direct Preference Optimization (DPO) involves teaching a language model to identify which answers are preferred by humans. When a model generates multiple responses to the same query, humans compare them and indicate which one is better. For instance, given the prompt "How do I reset my password?", one answer might be clear and helpful (Answer A), while another could be lengthy and confusing (Answer B). A human can simply state that Answer A is superior to Answer B.

DPO utilizes numerous such human preferences during the training process. It adjusts the model to make it more likely to produce responses similar to the preferred answers. However, the quality of DPO's output depends on the accuracy, consistency, and fairness of the human judgments provided.

Reinforcement Learning from Human Feedback (RLHF) shares similarities with DPO, as it also utilizes human preferences to teach a model which responses are more favorable. The primary difference lies in the training process. In classic RLHF, there are three stages:

1. Begin with a base model: Start with a pre-trained language model capable of generating text, but potentially lacking consistency in following instructions. This model is then fine-tuned on examples of good conversations and instructions.

2. Train a reward model: Humans compare different answers generated by the model. For example, when asked to explain photosynthesis, one answer might be straightforward (Answer A), while another could be more technical (Answer B). Humans then rate which answer they prefer. A separate model, the reward model, is trained on these preferences, learning to predict how much a human would like a particular response. Instead of simply stating that Answer A is better than Answer B, the reward model assigns scores to each response.

3. Practice against the reward model: The language model generates new responses, and the reward model assigns scores to them. The language model is then updated using reinforcement learning to produce responses that receive higher rewards. This process can be visualized as the language model generating an answer, the reward model scoring it, and reinforcement learning updating the language model accordingly.

A constraint is often included in the RLHF process to prevent the updated model from deviating too far from its original behavior. This constraint is typically implemented using a KL-divergence penalty. Without such constraints, the model might find unconventional ways to increase its reward while generating subpar responses.

To illustrate this concept, consider a student learning to write essays. Initially, a group of teachers evaluates various essays and determines which ones are superior. Next, a grading assistant is trained to mimic those teachers' judgments. Now, when the student writes a new essay, they receive a score from the grading assistant, which serves as feedback for improvement. The teachers do not need to grade every essay; the grading assistant provides continuous feedback as the student practices.

The key distinction between DPO and RLHF lies in how human preferences are utilized following their input. In RLHF, the preferences are first employed to train a separate reward model, which the language model then optimizes against using reinforcement learning. Conversely, DPO directly employs the preference pairs to optimize the language model without a separate reward model and RL loop. This distinction can be likened to a simplified vs. more complex training pipeline.