You Thought Training an AI Model Was Enough. Then There's Post-Training

Rijul is developing LiveReview, an AI code review system designed to assess the potential impact of code changes on business-critical systems. Training an AI model to become a helpful assistant involves more than just feeding it large amounts of data. Post-training techniques are necessary to ensure the model responds appropriately to questions and follows instructions.

Supervised Fine-Tuning (SFT) is one such technique where the model is shown thousands of example responses crafted by humans or strong existing models. The goal is to teach the model to produce the expected format, follow instructions, and respond helpfully. However, this method can be limited by the quality of the provided responses, especially when dealing with a vast number of potential questions.

An alternative approach is Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO). Both techniques rely on teaching the model to prefer certain responses over others. In DPO, the model is directly tuned using preference data, whereas PPO uses a separate reward model to predict the quality of responses before optimizing the language model.

DPO is often easier to implement and can be more cost-effective than PPO, as it eliminates the need for a separate reward model during the preference optimization process.